DJ Spoofer - Django + Web Scraping Made Easy
Setup Local Dev Environment
Setup SSH Keys
Generate SSH Keys
Run command:
ssh-keygen -t rsa -b 4096 -C "your_email@domain.com"
Hit enter to accept default file location, and again to accept empty passphrase
Copy Public Key
Install xclip
sudo apt-get install xclip
To copy to clipboard, run:
xclip -sel clip < ~/.ssh/id_rsa.pub
Install IntelliJ Toolbox
Download tarball from HERE
Extract tarball and replace <VERSION>
with the actual package name:
sudo tar -xzf jetbrains-toolbox-<VERSION>.tar.gz -C /opt
Run the install script. Replace <VERSION>
with the actual package name:
cd /opt/jetbrains-toolbox-<VERSION>
./jetbrains-toolbox
Follow prompts and install Pycharm Community IDE
Install Curl
sudo snap install curl
Install python-venv
sudo apt install python3.8-venv
Install distutils
sudo apt install python3-distutils
Install Poetry
sudo curl -sSL https://install.python-poetry.org | python3 -
Add Poetry Env Variable
Append the following to ~/.profile
export PATH="/home/adrian/.local/bin:$PATH
Test the poetry install:
poetry --version
Pull Dev Repo
Create project directory and initialize Git
git init
Add Remote Origin
git remote add origin git@github.com:adrianmeraz/marz-explorer.git
Pull repo
git pull
Create new branch and switch to branch
git checkout -b <BRANCH_NAME>
Delete local branch
git branch -d <BRANCH_NAME>
Add .env file
Use .env example to set environment variables.
NOTE
When deploying to remote environments, the environment variables must be set in Git secrets, as well as the lambda deployment.
Verify Github Action Secret
DATABASE_URL
should be set to postgres://postgres:postgres@localhost:5432/postgres
This should match values in the github actions yml
Install Poetry
Install poetry with instructions HERE
Verify the installation with a version check
poetry --version
If command not found, set the path variable:
source $HOME/.poetry/env
Install Dependencies
poetry install
Generate the secret key
import secrets
print(secrets.token_urlsafe())
Add to .env file as SECRET_KEY environment variable
Local Database Setup
Setup Postgres Local Instance (Ubuntu)
sudo apt update
sudo apt install postgresql postgresql-client
Check Instance Status
systemctl status postgresql.service
Change System User Password
NOTE: Don't use the @ symbol in the password!
sudo su - postgres
psql -c "alter user postgres with password '<DB_PASSWORD>'"
Start PSQL Shell
psql
Get Connection Details
\conninfo
Details should look similar to this:
You are connected to database "postgres" as user "postgres" via socket in "/var/run/postgresql" at port "5432".
Create Local Postgres Database
Replace placeholder values with real values
sudo -u postgres psql
postgres=# create database <DB_NAME>;
postgres=# create user <DB_USERNAME> with encrypted password '<DB_PASSWORD>';
postgres=# grant all privileges on database <DB_NAME> to <DB_USERNAME>;
postgres=# ALTER USER <DB_USERNAME> CREATEDB;
To Delete the local database, run:
postgres=# DROP DATABASE <DB_NAME>;
Local jdbc string
postgres://<DB_USERNAME>:<DB_PASSWORD>@localhost:5432/<DB_NAME>
Create Database on Existing Instance using admin command (Optional)
The initial database url <DB_INSTANCE_URL> will be:
postgres://postgres:<PASSWORD>@localhost:5432/postgres
Run the command from the project root to create the database and service user
poetry run python manage.py create_db --db-url="<DB_INSTANCE_URL>" --db-name="<NEW_DB_NAME>" --db-username="<NEW_DB_USERNAME>" --db-password="<NEW_DB_PASSWORD>"
Initialize Database
poetry run python manage.py makemigrations
git add .
poetry run python manage.py migrate
poetry run python manage.py createsuperuser
Create Super User
Set the following environment variables:
DJANGO_SUPERUSER_EMAIL DJANGO_SUPERUSER_USERNAME DJANGO_SUPERUSER_PASSWORD
poetry run python manage.py makemigrations
git add .
poetry run zappa manage <STAGE_NAME> "migrate"
poetry run zappa manage <STAGE_NAME> "createsuperuser --noinput"
Admin Commands
Poetry Utilities
Create Initial Poetry File
poetry init
Install All Requirements
poetry install
Update all Poetry Packages
poetry update
Update Individual Poetry Package
poetry update <PACKAGE_NAME>
View All Requirements
poetry show --tree
List Virtual Envs
poetry env list
Get Poetry Env Info
poetry env info
Remove Virtual Env
poetry env remove <ENV_NAME>
Clear Cache
poetry cache clear . --all
Poetry Executable Path
which poetry
Database Utilities
Migrations
After any Model Updates:
- Create migrations
- Add migrations to git
- Migrate to run DB changes
poetry run python manage.py makemigrations
git add .
poetry run python manage.py migrate
poetry run python manage.py createsuperuser
Testing
Specify settings with --settings
Running tests
poetry run python manage.py test --settings=config.settings.local --no-input --parallel
To run against a single module, add the module name:
poetry run python manage.py test djspoofer.tests.test_views --settings=config.settings.local --no-input --parallel
Faster Tests
NOTE Tests must be isolated and able to run independently! The --parallel
flag causes incorrect values
coverage run manage.py test --settings=config.settings.local --keepdb
Generate Coverage Report
Must be run after coverage tests have been run
coverage report
Generate Coverage HTML Report
coverage html
This creates the directory coverage html. Open the index.html to see the full report
Coverage Issues
If the numbers reported aren't what's expected, or tests are missing, verify that an empty __init__.py
is in the tests
directory for each app.
Contributions
Contributions are encouraged and welcome!
The general steps to contribute are:
- Create an issue for a feature/bug, if not already created
- Create a feature branch with the issue in the name
- Example: If the issue is
#123
, create the branch asfeature/DC-123
- Example: If the issue is
- Write Tests
- Write Code - All code must have tests and cannot lower coverage!
- Verify all tests pass
- Pull all and merge all changes from the dev branch
- Create a pull request to merge into the dev branch