Standard Operating Procedures

Overview

This document outlines the standard operating procedures (SOPs) for managing and maintaining the infrastructure and projects. Each section provides step-by-step instructions to ensure consistent and reliable operations.

Table of Contents

Server Updates

Purpose

To ensure that all servers are up-to-date with the latest security patches and software updates.

Frequency

  • Critical Security Updates: As soon as they are released.
  • Regular Updates: Weekly or as scheduled.

Procedure

  1. Access the Server:

  2. SSH into the server: bash ssh user@your-server-ip

  3. Update Package Lists:

  4. Run the following command to update the package lists: bash sudo apt-get update

  5. Upgrade Installed Packages:

  6. To upgrade all installed packages, use: bash sudo apt-get upgrade

  7. Reboot If Necessary:

  8. If any kernel or critical system files were updated, a reboot may be necessary: bash sudo reboot

  9. Verification:

  10. After the server restarts, verify that all services are running as expected: bash sudo systemctl status <service-name>

  11. Log the Update:

  12. Document the update in your maintenance log, noting the date, time, and any issues encountered.

SSL Certificate Renewals

To renew the certificate, please advise with SSL Renewal Guide.

Purpose

To renew SSL certificates before they expire to maintain secure HTTPS connections.

Frequency

  • Certificate Expiry Check: Every 3 months.
  • Renewal: At least 20 days before expiration.

Procedure

  1. Check Expiry Date:
  2. To check the SSL certificate expiry date, use:
openssl s_client -servername domain.* -connect domain.*:443 | openssl x509 -noout -dates
  1. Renew Certificate:
  2. If using Let's Encrypt:
# method 1
certbot renew
# method 2
certbot renew --cert-name domain.*
# method 3 (Usually Works)
# For this a file must be created at the root dir of domain like: /var/www/bigbluebutton-default/.well-known/acme-challenge/*filename* which will be given by certbot
certbot certonly --agree-tos -d vclass2.sahostyle.com --manual -m erfan226@gmail.com
# method 4 (Usually Works)
# This one needs change of DNS from top-level domain
certbot certonly --agree-tos -d vclass2.sahostyle.com --manual -m erfan226@gmail.com --preferred-challenges dns 

Note on DNS challenge: When asked yes, copy the given DNS record (ACME TXT Challenge) and replace it in both CPanel and CloudFlare/Host then continue the renewal (Don’t need to wait much for the dns to take effect, it takes 5 minutes at most):

  1. Verify the Renewal:
  2. Check the new expiry date to confirm the renewal was successful:
openssl s_client -servername domain.* -connect domain.*:443 | openssl x509 -noout -dates

Restart Nginx

  1. Restart Web Server: In case an error occurs, running the following command might help:
sudo systemctl restart systemd-resolved.service
  • Then estart the web server to apply the new certificate:
sudo systemctl restart nginx

or

sudo systemctl restart apache2
  1. Log the Renewal:
  2. Document the renewal in your maintenance log.

Database Backups

Purpose

To ensure regular backups of the database to prevent data loss.

Frequency

  • Full Backup: Weekly.
  • Incremental Backup: Daily.

Procedure

  1. Access the Database Server:
  2. SSH into the database server:
ssh user@database-server-ip
  1. Backup the Database:
  2. For MySQL/MariaDB:
mysqldump -u username -p database_name > /path/to/backup/database_name_$(date +%F).sql
  • For PostgreSQL:
pg_dump -U username -F c database_name > /path/to/backup/database_name_$(date +%F).backup
  1. Verify the Backup:

  2. Check the backup file size and try restoring to a test environment to ensure integrity.

  3. Transfer Backup to Secure Storage:

  4. Securely transfer the backup to a remote storage location:
scp /path/to/backup/database_name_$(date +%F).sql user@backup-server:/path/to/storage/
  1. Log the Backup:
  2. Document the backup in your maintenance log.

User Management

Purpose

To manage user accounts on the server and ensure only authorized personnel have access.

Frequency

  • User Review: Quarterly.
  • Account Creation/Deletion: As needed.

Procedure

  1. Add a New User:
  2. Create a new user:
sudo adduser newuser
  • Add the user to the sudo group if necessary:
sudo usermod -aG sudo newuser
  1. Remove a User:
  2. Delete a user and their home directory:
sudo deluser --remove-home username
  1. Review User Accounts:
  2. List all users and review:
cat /etc/passwd
  1. Log Changes:
  2. Document any user additions, deletions, or modifications.

Incident Response

Purpose

To outline steps to take during a security incident or service outage.

Frequency

  • Incident Response: As needed.

Procedure

  1. Identify the Incident:

  2. Determine the type and scope of the incident (e.g., DDoS, unauthorized access).

  3. Contain the Incident:

  4. Isolate affected systems to prevent further damage.

  5. Notify the Team:

  6. Inform relevant team members about the incident.

  7. Investigate and Remediate:

  8. Conduct a thorough investigation to identify the root cause.

  9. Apply patches, updates, or other fixes as necessary.

  10. Recover Systems:

  11. Restore systems from backups if necessary and verify functionality.

  12. Document the Incident:

  13. Complete an incident report documenting the details, response actions, and lessons learned.