The Pixel District

Build an E-Commerce Site with Wagtail CMS, Bootstrap & Django Framework.

Paul Kiragu — Sun, 06 Feb 2022 18:21:18 GMT

In this blog, we build a fully functional e-commerce site with Django and Wagtail CMS. I will also be briefly discussing:

Why choose django framework to build a custom e-commerce website
Which One is Better For Your Website? Django CMS Vs WordPress

You can check the demo of the final product at the following URL

Why choose Django for e-commerce?

First, here are some Django features to consider if you're looking for the right framework to build a shop.

→ Scalability

Django is perfect for e-commerce startups, as it's a good fit for small websites and can scale perfectly with business growth. You can rely on Django to handle hundreds/thousands of visitors at a time. It's built with independent components you can unplug or replace depending on your needs at any specific time.

→ Security

With e-commerce, you want to make sure merchants and clients alike feel safe through your shopping experience. Django prevents a whole lot of common security mistakes, often weakening traditional PHP CMSs. For instance, Django hides your site's source code from direct viewing on the web by dynamically generating web pages.

→ Feature-rich

Compared to most frameworks, Django comes with way more features out-of-the-box. It allows you to build an app right off the bat. Perfect for supporting your online store with functionalities such as user authentification, content management, or RSS feed. If something seems to be missing, you can count on Django's community and plugins ecosystem to extend your app!

→ SEO-friendly

SEO is paramount for any online business. Django advocates best practices for SEO. Human-readable URLs and sitemap features are sure to please any marketing team.

Oh, and also, it's fast, which is always great for both customer experience and SEO.

→ Reliable

Django makes it easy to build custom e-commerce websites. It has a modern, easy-to-use Python Web Framework that encourages rapid development and scalable architecture. Its high level abstraction make it easier to build, connect, configure, and maintain applications. In addition, Django makes it easy to add caching, secure authentication, and URL-agnostic views

Which One is Better For Your E-commerce Website? Django CMS Vs WordPress

Django database API makes it easy to work with databases. Django has a robust and secure authentication system. Django has a built in template engine, which means that you do not need to use a separate template language. Django is a free, open source web framework to! Django also follows the DRY(Don't Repeat Yourself) principle, DRY is the principle of reducing repetition in the code, referring back to a single source—or "snippet"—of reusable code whenever you need it. Django also removes a lot of overhead. Develiopers do not have to worry about making their programs compatible with different versions of Python, which means they can spend more time about the application.

Wagtail CMS + Snipcart e-commerce

Wagtail is a developer-first Django content management system. Free and open-source, the good-hearted folks at Torchbox developed it. It's elegant, flexible, and, IMHO, kicks ass.

In the following Wagtail tutorial, the CMS will be in charge of creating and managing products that users will then be able to buy through a shopping cart.

By the end of it, you'll have a solid Django-powered e-commerce site up and running.

Below is a screenshot of the shop we are going to build:

Shall we begin!

Pre-requisites

A Snipcart account.

1. Creating a Wagtail app

Use pip, which is packaged with Python, to install Wagtail and its dependencies:

$ python3 -m  venv venv
$ source venv/bin/activate
$ pip install wagtail requests

Open a terminal and launch a new Wagtail site:

$ wagtail start snipcartwagtaildemo
$ cd snipcartwagtaildemo

We have an extra step to complete the Wagtail setup, and it's to install the wagtail.contrib.settings plugin that we'll require later on.

In your new Wagtail project, open the base.py file located in snipcartwaigtaildemo/settings folder. Then, add wagtail.contrib.settings to the INSTALLED_APPS array.

# ./setting/base.py
INSTALLED_APPS = [
    ...,
    'wagtail.contrib.settings'
]

1.1 Models Creation.

The first thing you need to do is create your Page models. Wagtail uses these Django models to generate a page type.

Open the models.py file located in the home folder of your product. This is where you'll define all your custom models.

Create two different models:

Product: defines the product you're selling.

ProductCustomField: defines a single product custom field.

Let's begin by importing the required modules:

# ./home/models.py

from django.db import models

from wagtail.core.models import Page

from modelcluster.fields import ParentalKey

from wagtail.core.models import Page, Orderable
from wagtail.admin.edit_handlers import FieldPanel, MultiFieldPanel, InlinePanel
from wagtail.images.edit_handlers import ImageChooserPanel
from wagtail.contrib.settings.models import BaseSetting, register_setting
from django.core.paginator import EmptyPage, PageNotAnInteger, Paginator
from django.template.response import TemplateResponse
from django.core.paginator import Paginator

Now add the Product model:

# ./home/models.py

class Product(Page):
    def get_context(self, request):
        context = super().get_context(request)
        fields = []
        for f in self.custom_fields.get_object_list():
            if f.options:
                f.options_array = f.options.split('|')
                fields.append(f)
            else:
                fields.append(f)

        context['custom_fields'] = fields

        return context
    sku = models.CharField(max_length=255)
    short_description = models.TextField(blank=True, null=True)
    price = models.DecimalField(decimal_places=2, max_digits=10)
    image = models.ForeignKey(
        'wagtailimages.Image',
        null=True,
        blank=True,
        on_delete=models.SET_NULL,
        related_name='+'
    )

    content_panels = Page.content_panels + [
        FieldPanel('sku'),
        FieldPanel('price'),
        ImageChooserPanel('image'),
        FieldPanel('short_description'),
        InlinePanel('custom_fields', label='Custom fields'),
    ]

Update the homepage class which is responsible for listing products on the homepage only. I will be listing only 6 products on the homepage. In any Wagtail Page, you can override a method name get_context. You can add the data that the view will receive in parameters

class HomePage(Page):
    def get_context(self, request):
        context = super().get_context(request)

        context['products'] = Product.objects.child_of(self).live()[:6]

        return context

And ProductCustomField:

# ./home/models.py

class ProductCustomField(Orderable):
    product = ParentalKey(Product, on_delete=models.CASCADE, related_name='custom_fields')
    name = models.CharField(max_length=255)
    options = models.CharField(max_length=500, null=True, blank=True)

    panels = [
        FieldPanel('name'),
        FieldPanel('options')
    ]

We need to add one more view that will be responsible for showing all products in our online shop, and for paginating the page. I will be paginating on the backend side as loading all the objects on a page is not very efficient. We can however cache our data and load it all on the front end. This will make the pagination seamless, this all depends on the size of your database.

def shop(request):
    products = Product.objects.live()
    p = Paginator(products, 10)
    #shows number of items in page
    totalProducts = (p.count)
    pageNum = request.GET.get('page', 1)
    page1 = p.page(pageNum)

    return TemplateResponse(request, 'home/shop.html', {
        'products': products,
        'dataSaved':page1
    })

Finally, add the about us and contact us section views, like so:

def about(request):
    return TemplateResponse(request, 'home/about.html', {
    }) 

def contact(request):
    return TemplateResponse(request, 'home/contact.html', {
    })

2. Adding Snipcart configuration settings

Let's make sure you can update the Snipcart API key directly from Wagtail's dashboard.

You'll need to add site settings to do so.

Site settings are special fields that you can add to your models.py file. They'll appear in the Wagtail Settings section of the dashboard.

Import this module:

# ./home/models.py

from wagtail.contrib.settings.models import BaseSetting, register_setting

Then add these:

# ./home/models.py

@register_setting
class SnipcartSettings(BaseSetting):
    api_key = models.CharField(
        max_length=255,
        help_text='Your Snipcart public API key'
    )

3. Database migrations

Now that your models are created, you'll need to generate database migrations and run them.

In your terminal, use the makemigrations command:

$ python3 manage.py makemigrations
$ python3 manage.py migrate

Finally, create your first CMS user with the createsuperuser command:

$ python3 manage.py createsuperuser

It will ask for an email address, a username, and to create a password. Don't forget the username and password you picked; you will need them to log into Wagtail's dashboard.

4. Creating products

Fire up your server like so:

$ python3 manage.py runserver

Now open your browser and navigate to http://127.0.0.1:8000/admin. Use the credentials you set up earlier to log in.

Select the Home page in Wagtail's menu. Then click on the Add child page button.

You'll be asked to pick a type of page, select Home page and give it a name.

Go back to the Home page menu and click Add child page button under your newly created page.

Once again, you'll be asked to pick a type of page, this time select Product.

You can create as many products as you wish.

4.1 Adding Snipcart API key

Remember the SnipcartSettings class you created? You'll be able to configure your API key by expanding the Settings menu and going to Snipcart settings.

Open Snipcart's dashboard and get your public API key (Test or Live), go back to Wagtail, and paste it in the API key field.

Save your settings.

Adding new URL routes

I have barely used wagtail and i am not very familiar with how the URL's are handled but i created my own URL for the shop view, for listing all products:

from django.conf import settings
from django.urls import include, path
from django.contrib import admin

from wagtail.admin import urls as wagtailadmin_urls
from wagtail.core import urls as wagtail_urls
from wagtail.documents import urls as wagtaildocs_urls

from search import views as search_views
from home import models as p
urlpatterns = [
    path('django-admin/', admin.site.urls),

    path('admin/', include(wagtailadmin_urls)),
    path('documents/', include(wagtaildocs_urls)),

    path('search/', search_views.search, name='search'),
    path('shop/', p.shop, name='shop'),
    path('about/', p.about, name='about'),
    path('contact/', p.contact, name='contact'),

]


if settings.DEBUG:
    from django.conf.urls.static import static
    from django.contrib.staticfiles.urls import staticfiles_urlpatterns

    # Serve static and media files from development server
    urlpatterns += staticfiles_urlpatterns()
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

urlpatterns = urlpatterns + [
    # For anything not caught by a more specific rule above, hand over to
    # Wagtail's page serving mechanism. This should be the last pattern in
    # the list:
    path("", include(wagtail_urls)),

    # Alternatively, if you want Wagtail pages to be served from a subpath
    # of your site, rather than the site root:
    #    path("pages/", include(wagtail_urls)),
]

5. Templating

Let us now proceed with creating our HTML files and modifying our application UI. The template is from this Colorlib URL.

Homepage

On our homepage, we need to display a few items from our database. We already specified 6 items on our homepage view. Create 3 files on the home/templates/home folder:

home_page.html
product.html
shop.html

Before i display the whole code, let me explain a few important bits, starting with the head of the HTML files

{% load static wagtailcore_tags wagtailuserbar %}
{% load wagtailimages_tags %}
html>
<html lang="en">

<head>
  <title>Tafari Pharmacytitle>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

  {% load static wagtailsettings_tags %}
    {% get_settings %}

    {# Global stylesheets #}
    <link href="https://fonts.googleapis.com/css?family=Rubik:400,700|Crimson+Text:400,400i" rel="stylesheet">
    <link rel="stylesheet" href="{% static 'fonts/icomoon/style.css' %}">

    <link rel="stylesheet" href="{% static 'css/bootstrap.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/magnific-popup.css' %}">
    <link rel="stylesheet" href="{% static 'css/jquery-ui.css' %}">
    <link rel="stylesheet" href="{% static 'css/owl.carousel.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/owl.theme.default.min.css' %}">


    <link rel="stylesheet" href="{% static 'css/aos.css' %}">

    <link rel="stylesheet" href="{% static 'css/style.css' %}">

    {# Snipcart #}

    {% if settings.home.SnipcartSettings.api_key %}
  <script async src="https://cdn.snipcart.com/themes/v3.2.1/default/snipcart.js">script>
  <div hidden id="snipcart" data-api-key="{{ settings.home.SnipcartSettings.api_key }}">div>
  <link rel="stylesheet" href="https://cdn.snipcart.com/themes/v3.2.1/default/snipcart.css" />
    {% endif %}

head>

We have the reference for the snipcart tags where it detects the API key we added on the settings file and have reference for the snipcart files as cdn files.

<div class="row">
          {% for product in products %}
          <div class="col-sm-6 col-lg-4 text-center item mb-4">
            <a href="{{ product.get_url }}">
                {% image product.image fill-370x270 as tmp_image %}
                <img src="{{ tmp_image.url }}" alt="Image">
            </a>
            <h3 class="text-dark"><a href="{{ product.get_url }}">{{ product.title }}</a></h3>
            <p class="price">Ksh. {{ product.price }}</p>
          </div>
          {% endfor %}
        </div>

We also have the product part that loops through the products and displays them all. For more details, you can visit the snip cart docs. The whole code for the homepage is:

{% load static wagtailcore_tags wagtailuserbar %}
{% load wagtailimages_tags %}
html>
<html lang="en">

<head>
  <title>Tafari Pharmacytitle>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

  {% load static wagtailsettings_tags %}
    {% get_settings %}

    {# Global stylesheets #}
    <link href="https://fonts.googleapis.com/css?family=Rubik:400,700|Crimson+Text:400,400i" rel="stylesheet">
    <link rel="stylesheet" href="{% static 'fonts/icomoon/style.css' %}">

    <link rel="stylesheet" href="{% static 'css/bootstrap.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/magnific-popup.css' %}">
    <link rel="stylesheet" href="{% static 'css/jquery-ui.css' %}">
    <link rel="stylesheet" href="{% static 'css/owl.carousel.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/owl.theme.default.min.css' %}">


    <link rel="stylesheet" href="{% static 'css/aos.css' %}">

    <link rel="stylesheet" href="{% static 'css/style.css' %}">

    {# Snipcart #}

    {% if settings.home.SnipcartSettings.api_key %}
  <script async src="https://cdn.snipcart.com/themes/v3.2.1/default/snipcart.js">script>
  <div hidden id="snipcart" data-api-key="{{ settings.home.SnipcartSettings.api_key }}">div>
  <link rel="stylesheet" href="https://cdn.snipcart.com/themes/v3.2.1/default/snipcart.css" />
    {% endif %}

head>

<body>
    {% wagtailuserbar %}
  <div class="site-wrap">


    <div class="site-navbar py-2">

      <div class="search-wrap">
        <div class="container">
          <a href="#" class="search-close js-search-close"><span class="icon-close2">span>a>
          <form action="#" method="post">
            <input type="text" class="form-control" placeholder="Search keyword and hit enter...">
          form>
        div>
      div>

      <div class="container">
        <div class="d-flex align-items-center justify-content-between">
          <div class="logo">
            <div class="site-logo">
              <a href="/" class="js-logo-clone">Tafari Pharmacya>
            div>
          div>
          <div class="main-nav d-none d-lg-block">
            <nav class="site-navigation text-right text-md-center" role="navigation">
              <ul class="site-menu js-clone-nav d-none d-lg-block">
                <li class="active"><a href="/">Homea>li>
                {% if request.path == '/' %}
                <li><a href="/shop">Storea>li>
                {% endif %}
                <li class="has-children">
                  <a href="#">Medicationsa>
                  <ul class="dropdown">
                    <li><a href="#">Supplementsa>li>
                    <li class="has-children">
                      <a href="#">Vitaminsa>
                      <ul class="dropdown">
                        <li><a href="#">Supplementsa>li>
                        <li><a href="#">Vitaminsa>li>
                        <li><a href="#">Diet & Nutritiona>li>
                        <li><a href="#">Tea & Coffeea>li>
                      ul>
                    li>
                    <li><a href="#">Diet & Nutritiona>li>
                    <li><a href="#">Tea & Coffeea>li>

                  ul>
                li>
                <li><a href="/about">Abouta>li>
                <li><a href="/contact">Contacta>li>
              ul>
            nav>
          div>
          <div class="icons">
            <a href="#" class="icons-btn d-inline-block js-search-open"><span class="icon-search">span>a>
            <a href="" class="icons-btn d-inline-block bag">
              <span class="snipcart-checkout icon-shopping-bag">span>
              <span class="snipcart-checkout number snipcart-items-count">span>
            a>
            <a href="#" class="site-menu-toggle js-menu-toggle ml-3 d-inline-block d-lg-none"><span
                class="icon-menu">span>a>
          div>
        div>
      div>
    div>

    <div class="site-blocks-cover" style="background-image: url('{% static 'images/hero_1.jpg' %}');">
      <div class="container">
        <div class="row">
          <div class="col-lg-7 mx-auto order-lg-2 align-self-center">
            <div class="site-block-cover-content text-center">
              <h2 class="sub-title">Effective Medicine, New Medicine Everydayh2>
              <h1>Welcome To Tafari Pharmacyh1>
              <p>
                {% if request.path == '/' %}
                <a href="#" class="btn btn-primary px-5 py-3">Shop Nowa>
                {% endif %}
              p>
            div>
          div>
        div>
      div>
    div>

    <div class="site-section">
      <div class="container">
        <div class="row align-items-stretch section-overlap">
          <div class="col-md-6 col-lg-4 mb-4 mb-lg-0">
            <div class="banner-wrap bg-primary h-100">
              <a href="#" class="h-100">
                <h5>Free <br> Deliveryh5>
                <p>
                  Amet sit amet dolor
                  <strong>Lorem, ipsum dolor sit amet consectetur adipisicing.strong>
                p>
              a>
            div>
          div>
          <div class="col-md-6 col-lg-4 mb-4 mb-lg-0">
            <div class="banner-wrap h-100">
              <a href="#" class="h-100">
                <h5>Season <br> Sale 50% Offh5>
                <p>
                  Amet sit amet dolor
                  <strong>Lorem, ipsum dolor sit amet consectetur adipisicing.strong>
                p>
              a>
            div>
          div>
          <div class="col-md-6 col-lg-4 mb-4 mb-lg-0">
            <div class="banner-wrap bg-warning h-100">
              <a href="#" class="h-100">
                <h5>Buy <br> A Gift Cardh5>
                <p>
                  Amet sit amet dolor
                  <strong>Lorem, ipsum dolor sit amet consectetur adipisicing.strong>
                p>
              a>
            div>
          div>

        div>
      div>
    div>

    <div class="site-section">
      <div class="container">
        <div class="row">
          <div class="title-section text-center col-12">
            <h2 class="text-uppercase">Popular Productsh2>
          div>
        div>


        <div class="row">
          {% for product in products %}
          <div class="col-sm-6 col-lg-4 text-center item mb-4">
            <a href="{{ product.get_url }}">
                {% image product.image fill-370x270 as tmp_image %}
                <img src="{{ tmp_image.url }}" alt="Image">
            a>
            <h3 class="text-dark"><a href="{{ product.get_url }}">{{ product.title }}a>h3>
            <p class="price">Ksh. {{ product.price }}p>
          div>
          {% endfor %}
        div>


        <div class="row mt-5">
          <div class="col-12 text-center">
            {% if request.path == '/' %}
            <a href="/shop" class="btn btn-primary px-4 py-3">View All Productsa>
            {% endif %}
          div>
        div>
      div>
    div>


    <div class="site-section bg-light">
      <div class="container">
        <div class="row">
          <div class="title-section text-center col-12">
            <h2 class="text-uppercase">New Productsh2>
          div>
        div>
        <div class="row">
          <div class="col-md-12 block-3 products-wrap">
            <div class="nonloop-block-3 owl-carousel">

              <div class="text-center item mb-4">
                <a href="shop-single.html"> <img src="{% static 'images/product_03.png' %}" alt="Image">a>
                <h3 class="text-dark"><a href="shop-single.html">Umcka Cold Carea>h3>
                <p class="price">$120.00p>
              div>

              <div class="text-center item mb-4">
                <a href="shop-single.html"> <img src="{% static 'images/product_01.png' %}" alt="Image">a>
                <h3 class="text-dark"><a href="shop-single.html">Umcka Cold Carea>h3>
                <p class="price">$120.00p>
              div>

              <div class="text-center item mb-4">
                <a href="shop-single.html"> <img src="{% static 'images/product_02.png' %}" alt="Image">a>
                <h3 class="text-dark"><a href="shop-single.html">Umcka Cold Carea>h3>
                <p class="price">$120.00p>
              div>

              <div class="text-center item mb-4">
                <a href="shop-single.html"> <img src="{% static 'images/product_04.png' %}" alt="Image">a>
                <h3 class="text-dark"><a href="shop-single.html">Umcka Cold Carea>h3>
                <p class="price">$120.00p>
              div>

            div>
          div>
        div>
      div>
    div>

    <div class="site-section">
      <div class="container">
        <div class="row">
          <div class="title-section text-center col-12">
            <h2 class="text-uppercase">Testimonialsh2>
          div>
        div>
        <div class="row">
          <div class="col-md-12 block-3 products-wrap">
            <div class="nonloop-block-3 no-direction owl-carousel">

              <div class="testimony">
                <blockquote>
                  <img src="{% static 'images/person_1.jpg' %}" alt="Image" class="img-fluid w-25 mb-4 rounded-circle">
                  <p>“Lorem ipsum dolor, sit amet consectetur adipisicing elit. Nemo omnis voluptatem consectetur quam tempore obcaecati maiores voluptate aspernatur iusto eveniet, placeat ab quod tenetur ducimus. Minus ratione sit quaerat unde.”p>
                blockquote>

                <p>— Kelly Holmesp>
              div>

              <div class="testimony">
                <blockquote>
                  <img src="{% static 'images/person_2.jpg' %}" alt="Image" class="img-fluid w-25 mb-4 rounded-circle">
                  <p>“Lorem ipsum dolor, sit amet consectetur adipisicing elit. Nemo omnis voluptatem consectetur quam tempore
                    obcaecati maiores voluptate aspernatur iusto eveniet, placeat ab quod tenetur ducimus. Minus ratione sit quaerat
                    unde.”p>
                blockquote>

                <p>— Rebecca Morandop>
              div>

              <div class="testimony">
                <blockquote>
                  <img src="{% static 'images/person_3.jpg' %}" alt="Image" class="img-fluid w-25 mb-4 rounded-circle">
                  <p>“Lorem ipsum dolor, sit amet consectetur adipisicing elit. Nemo omnis voluptatem consectetur quam tempore
                    obcaecati maiores voluptate aspernatur iusto eveniet, placeat ab quod tenetur ducimus. Minus ratione sit quaerat
                    unde.”p>
                blockquote>

                <p>— Lucas Gallonep>
              div>

              <div class="testimony">
                <blockquote>
                  <img src="{% static 'images/person_4.jpg' %}" alt="Image" class="img-fluid w-25 mb-4 rounded-circle">
                  <p>“Lorem ipsum dolor, sit amet consectetur adipisicing elit. Nemo omnis voluptatem consectetur quam tempore
                    obcaecati maiores voluptate aspernatur iusto eveniet, placeat ab quod tenetur ducimus. Minus ratione sit quaerat
                    unde.”p>
                blockquote>

                <p>— Andrew Neelp>
              div>

            div>
          div>
        div>
      div>
    div>

    <div class="site-section bg-secondary bg-image" style="background-image: url('{% static 'images/bg_2.jpg' %}');">
      <div class="container">
        <div class="row align-items-stretch">
          <div class="col-lg-6 mb-5 mb-lg-0">
            <a href="#" class="banner-1 h-100 d-flex" style="background-image: url('{% static 'images/bg_1.jpg' %}');">
              <div class="banner-1-inner align-self-center">
                <h2>Tafari Pharmacy Productsh2>
                <p>Lorem, ipsum dolor sit amet consectetur adipisicing elit. Molestiae ex ad minus rem odio voluptatem.
                p>
              div>
            a>
          div>
          <div class="col-lg-6 mb-5 mb-lg-0">
            <a href="#" class="banner-1 h-100 d-flex" style="background-image: url('{% static 'images/bg_2.jpg' %}');">
              <div class="banner-1-inner ml-auto  align-self-center">
                <h2>Rated by Expertsh2>
                <p>Lorem, ipsum dolor sit amet consectetur adipisicing elit. Molestiae ex ad minus rem odio voluptatem.
                p>
              div>
            a>
          div>
        div>
      div>
    div>


    <footer class="site-footer">
      <div class="container">
        <div class="row">
          <div class="col-md-6 col-lg-3 mb-4 mb-lg-0">

            <div class="block-7">
              <h3 class="footer-heading mb-4">About Ush3>
              <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Eius quae reiciendis distinctio voluptates
                sed dolorum excepturi iure eaque, aut unde.p>
            div>

          div>
          <div class="col-lg-3 mx-auto mb-5 mb-lg-0">
            <h3 class="footer-heading mb-4">Quick Linksh3>
            <ul class="list-unstyled">
              <li><a href="#">Supplementsa>li>
              <li><a href="#">Vitaminsa>li>
              <li><a href="#">Diet & Nutritiona>li>
              <li><a href="#">Tea & Coffeea>li>
            ul>
          div>

          <div class="col-md-6 col-lg-3">
            <div class="block-5 mb-5">
              <h3 class="footer-heading mb-4">Contact Infoh3>
              <ul class="list-unstyled">
                <li class="address">203 Fake St. Mountain View, San Francisco, California, USAli>
                <li class="phone"><a href="tel://23923929210">+2 392 3929 210a>li>
                <li class="email">emailaddress@domain.comli>
              ul>
            div>


          div>
        div>
        <div class="row pt-5 mt-5 text-center">
          <div class="col-md-12">
            <p>
              
              Copyright ©
              <script>document.write(new Date().getFullYear());script> All rights reserved | This template is made
              with <i class="icon-heart" aria-hidden="true">i> by <a href="https://colorlib.com" target="_blank"
                class="text-primary">Colorliba>
              
            p>
          div>

        div>
      div>
    footer>
  div>

  <script src="{% static 'js/jquery-3.3.1.min.js' %}">script>
  <script src="{% static 'js/jquery-ui.js' %}">script>
  <script src="{% static 'js/popper.min.js' %}">script>
  <script src="{% static 'js/bootstrap.min.js' %}">script>
  <script src="{% static 'js/owl.carousel.min.js' %}">script>
  <script src="{% static 'js/jquery.magnific-popup.min.js' %}">script>
  <script src="{% static 'js/aos.js' %}">script>

  <script src="{% static 'js/main.js' %}">script>

body>

html>

All Products Page

On the product.html, add the lines of code below. I didn't use the inbuilt django templates functionality to work in components so we have to repeat the tags we added in the homepage file, for every page we create. Copy paste the code below on the templates/home/shop.html:

{% load static wagtailcore_tags wagtailuserbar %}
{% load wagtailimages_tags %}
html>
<html lang="en">

<head>
  <title>Tafari Tafari Pharmacycy — Colorlib Templatetitle>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

  {% load static wagtailsettings_tags %}
    {% get_settings %}

    {# Global stylesheets #}
    <link href="https://fonts.googleapis.com/css?family=Rubik:400,700|Crimson+Text:400,400i" rel="stylesheet">
    <link rel="stylesheet" href="{% static 'fonts/icomoon/style.css' %}">

    <link rel="stylesheet" href="{% static 'css/bootstrap.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/magnific-popup.css' %}">
    <link rel="stylesheet" href="{% static 'css/jquery-ui.css' %}">
    <link rel="stylesheet" href="{% static 'css/owl.carousel.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/owl.theme.default.min.css' %}">


    <link rel="stylesheet" href="{% static 'css/aos.css' %}">

    <link rel="stylesheet" href="{% static 'css/style.css' %}">

    {# Snipcart #}

    {% if settings.home.SnipcartSettings.api_key %}
  <script async src="https://cdn.snipcart.com/themes/v3.2.1/default/snipcart.js">script>
  <div hidden id="snipcart" data-api-key="{{ settings.home.SnipcartSettings.api_key }}">div>
  <link rel="stylesheet" href="https://cdn.snipcart.com/themes/v3.2.1/default/snipcart.css" />
    {% endif %}

head>

<body>
    {% wagtailuserbar %}

  <div class="site-wrap">


    <div class="site-navbar py-2">

      <div class="search-wrap">
        <div class="container">
          <a href="#" class="search-close js-search-close"><span class="icon-close2">span>a>
          <form action="#" method="post">
            <input type="text" class="form-control" placeholder="Search keyword and hit enter...">
          form>
        div>
      div>

      <div class="container">
        <div class="d-flex align-items-center justify-content-between">
          <div class="logo">
            <div class="site-logo">
              <a href="/" class="js-logo-clone">Tafari Pharmacya>
            div>
          div>
          <div class="main-nav d-none d-lg-block">
            <nav class="site-navigation text-right text-md-center" role="navigation">
                <ul class="site-menu js-clone-nav d-none d-lg-block">
                  <li><a href="/">Homea>li>
                  <li class="active"><a href="/shop">Storea>li>
                  <li class="has-children">
                    <a href="#">Medicationsa>
                    <ul class="dropdown">
                      <li><a href="#">Supplementsa>li>
                      <li class="has-children">
                        <a href="#">Vitaminsa>
                        <ul class="dropdown">
                          <li><a href="#">Supplementsa>li>
                          <li><a href="#">Vitaminsa>li>
                          <li><a href="#">Diet & Nutritiona>li>
                          <li><a href="#">Tea & Coffeea>li>
                        ul>
                      li>
                      <li><a href="#">Diet & Nutritiona>li>
                      <li><a href="#">Tea & Coffeea>li>

                    ul>
                  li>
                  <li><a href="/about">Abouta>li>
                  <li><a href="/contact">Contacta>li>
                ul>
              nav>
          div>
          <div class="icons">
            <a href="#" class="icons-btn d-inline-block js-search-open"><span class="icon-search">span>a>
            <a href="" class="icons-btn d-inline-block bag">
                <span class="snipcart-checkout icon-shopping-bag">span>
                <span class="snipcart-checkout number snipcart-items-count">span>
              a>
            <a href="#" class="site-menu-toggle js-menu-toggle ml-3 d-inline-block d-lg-none"><span
                class="icon-menu">span>a>
          div>
        div>
      div>
    div>

    <div class="bg-light py-3">
      <div class="container">
        <div class="row">
          <div class="col-md-12 mb-0"><a href="/">Homea> <span class="mx-2 mb-0">/span> <strong class="text-black">Storestrong>div>
        div>
      div>
    div>

    <div class="site-section">
      <div class="container">



        {% for product in dataSaved.object_list %}
        <div class="row">
          <div class="col-sm-6 col-lg-4 text-center item mb-4">
            <a href="{{ product.get_url }}">
                {% image product.image fill-370x270 as tmp_image %}
                <img src="{{ tmp_image.url }}" alt="Image">
            a>
            <h3 class="text-dark"><a href="{{ product.get_url }}">{{ product.title }}a>h3>
            <p class="price">Ksh. {{ product.price }}p>
          div>
        div>
        {% endfor %}
        <div class="row mt-5">
          <div class="col-md-12 text-center">
            <div class="site-block-27">
                {% if dataSaved.has_other_pages %}
                <ul class="pagination justify-content-center">
                    {% if dataSaved.has_previous %}
                  <li class="page-item">
                    <a class="page-link" href="?page={{ dataSaved.previous_page_number }}" tabindex="-1">Previousa>
                  li>
                  {% endif %}

                  {% for i in dataSaved.paginator.page_range %}
                     {% if dataSaved.number == i %}
                  <li class="page-item active">
                      <a class="page-link" href="#">{{ i }}<span class="sr-only">(current)span>a>
                   li>
                  {% endif %}
                  {% endfor %} 
                  <li class="page-item">
                    {% if dataSaved.has_next %}
                    <a class="page-link" href="?page={{ dataSaved.next_page_number }}">Nexta>
                    {% endif %}
                  li>
                ul>
                {% endif %}
            div>
          div>
        div>
      div>
    div>


    <div class="site-section bg-secondary bg-image" style="background-image: url('{% static 'images/bg_2.jpg' %}');">
      <div class="container">
        <div class="row align-items-stretch">
          <div class="col-lg-6 mb-5 mb-lg-0">
            <a href="#" class="banner-1 h-100 d-flex" style="background-image: url('{% static 'images/bg_1.jpg' %}');">
              <div class="banner-1-inner align-self-center">
                <h2>Tafari Pharmacy Productsh2>
                <p>Lorem, ipsum dolor sit amet consectetur adipisicing elit. Molestiae ex ad minus rem odio voluptatem.
                p>
              div>
            a>
          div>
          <div class="col-lg-6 mb-5 mb-lg-0">
            <a href="#" class="banner-1 h-100 d-flex" style="background-image: url('{% static 'images/bg_2.jpg' %}');">
              <div class="banner-1-inner ml-auto  align-self-center">
                <h2>Rated by Expertsh2>
                <p>Lorem, ipsum dolor sit amet consectetur adipisicing elit. Molestiae ex ad minus rem odio voluptatem.
                p>
              div>
            a>
          div>
        div>
      div>
    div>


    <footer class="site-footer">
      <div class="container">
        <div class="row">
          <div class="col-md-6 col-lg-3 mb-4 mb-lg-0">

            <div class="block-7">
              <h3 class="footer-heading mb-4">About Ush3>
              <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Eius quae reiciendis distinctio voluptates
                sed dolorum excepturi iure eaque, aut unde.p>
            div>

          div>
          <div class="col-lg-3 mx-auto mb-5 mb-lg-0">
            <h3 class="footer-heading mb-4">Quick Linksh3>
            <ul class="list-unstyled">
              <li><a href="#">Supplementsa>li>
              <li><a href="#">Vitaminsa>li>
              <li><a href="#">Diet & Nutritiona>li>
              <li><a href="#">Tea & Coffeea>li>
            ul>
          div>

          <div class="col-md-6 col-lg-3">
            <div class="block-5 mb-5">
              <h3 class="footer-heading mb-4">Contact Infoh3>
              <ul class="list-unstyled">
                <li class="address">203 Fake St. Mountain View, San Francisco, California, USAli>
                <li class="phone"><a href="tel://23923929210">+2 392 3929 210a>li>
                <li class="email">emailaddress@domain.comli>
              ul>
            div>


          div>
        div>
        <div class="row pt-5 mt-5 text-center">
          <div class="col-md-12">
            <p>
              
              Copyright ©
              <script>document.write(new Date().getFullYear());script> All rights reserved | This template is made
              with <i class="icon-heart" aria-hidden="true">i> by <a href="https://colorlib.com" target="_blank"
                class="text-primary">Colorliba>
              
            p>
          div>

        div>
      div>
    footer>
  div>

  <script src="{% static 'js/jquery-3.3.1.min.js' %}">script>
  <script src="{% static 'js/jquery-ui.js' %}">script>
  <script src="{% static 'js/popper.min.js' %}">script>
  <script src="{% static 'js/bootstrap.min.js' %}">script>
  <script src="{% static 'js/owl.carousel.min.js' %}">script>
  <script src="{% static 'js/jquery.magnific-popup.min.js' %}">script>
  <script src="{% static 'js/aos.js' %}">script>

  <script src="{% static 'js/main.js' %}">script>
  <script>
    document.addEventListener('DOMContentLoaded', function() {
        document.querySelector('.custom-field-select').onchange = function(event) {
            if (event.target.dataset.field) {
                document.querySelector('.snipcart-add-item')
                    .dataset['itemCustom' + event.target.dataset.field + 'Value'] = event.target.value;
            }
        };
    },false);
script>
body>

html>

The code is the same as the home page code, the only difference is in the shop.html file, we paginate our webpage, and display all products instead of just 6 like in the homepage.

Product details and cart display page

For the products page, we need to display the product when a user clicks on an item, like so:

In order to do this, we need to create a template that will be associated with the Product page model. On the product.html in the home/templates/home folder, add:

{% load static wagtailcore_tags wagtailuserbar %}
{% load wagtailimages_tags %}
html>
<html lang="en">

<head>
  <title>Tafari Pharmacy — Colorlib Templatetitle>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

  {% load static wagtailsettings_tags %}
    {% get_settings %}

    {# Global stylesheets #}
    <link href="https://fonts.googleapis.com/css?family=Rubik:400,700|Crimson+Text:400,400i" rel="stylesheet">
    <link rel="stylesheet" href="{% static 'fonts/icomoon/style.css' %}">

    <link rel="stylesheet" href="{% static 'css/bootstrap.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/magnific-popup.css' %}">
    <link rel="stylesheet" href="{% static 'css/jquery-ui.css' %}">
    <link rel="stylesheet" href="{% static 'css/owl.carousel.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/owl.theme.default.min.css' %}">


    <link rel="stylesheet" href="{% static 'css/aos.css' %}">

    <link rel="stylesheet" href="{% static 'css/style.css' %}">

    {# Snipcart #}

    {% if settings.home.SnipcartSettings.api_key %}
  <script async src="https://cdn.snipcart.com/themes/v3.2.1/default/snipcart.js">script>
  <div hidden id="snipcart" data-api-key="{{ settings.home.SnipcartSettings.api_key }}">div>
  <link rel="stylesheet" href="https://cdn.snipcart.com/themes/v3.2.1/default/snipcart.css" />
    {% endif %}

head>

<body>
    {% wagtailuserbar %}
  <div class="site-wrap">


    <div class="site-navbar py-2">

      <div class="search-wrap">
        <div class="container">
          <a href="#" class="search-close js-search-close"><span class="icon-close2">span>a>
          <form action="#" method="post">
            <input type="text" class="form-control" placeholder="Search keyword and hit enter...">
          form>
        div>
      div>

      <div class="container">
        <div class="d-flex align-items-center justify-content-between">
          <div class="logo">
            <div class="site-logo">
              <a href="/" class="js-logo-clone">Tafari Pharmacya>
            div>
          div>
          <div class="main-nav d-none d-lg-block">
            <nav class="site-navigation text-right text-md-center" role="navigation">
                <ul class="site-menu js-clone-nav d-none d-lg-block">
                  <li><a href="/">Homea>li>
                  <li class="active"><a href="/shop">Storea>li>
                  <li class="has-children">
                    <a href="#">Medicationsa>
                    <ul class="dropdown">
                      <li><a href="#">Supplementsa>li>
                      <li class="has-children">
                        <a href="#">Vitaminsa>
                        <ul class="dropdown">
                          <li><a href="#">Supplementsa>li>
                          <li><a href="#">Vitaminsa>li>
                          <li><a href="#">Diet & Nutritiona>li>
                          <li><a href="#">Tea & Coffeea>li>
                        ul>
                      li>
                      <li><a href="#">Diet & Nutritiona>li>
                      <li><a href="#">Tea & Coffeea>li>

                    ul>
                  li>
                  <li><a href="/about">Abouta>li>
                  <li><a href="/contact">Contacta>li>
                ul>
              nav>
          div>
          <div class="icons">
            <a href="#" class="icons-btn d-inline-block js-search-open"><span class="icon-search">span>a>
            <a href="" class="icons-btn d-inline-block bag">
              <span class="snipcart-checkout icon-shopping-bag">span>
              <span class="snipcart-checkout number snipcart-items-count">span>
            a>
            <a href="#" class="site-menu-toggle js-menu-toggle ml-3 d-inline-block d-lg-none"><span
                class="icon-menu">span>a>
          div>
        div>
      div>
    div>

    <div class="bg-light py-3">
      <div class="container">
        <div class="row">
          <div class="col-md-12 mb-0"><a href="/">Homea> <span class="mx-2 mb-0">/span> <a
              href="/shop">Storea> <span class="mx-2 mb-0">/span> <strong class="text-black">{{ page.title }}strong>div>
        div>
      div>
    div>

    <div class="site-section">
      <div class="container">
        <div class="row">
          <div class="col-md-5 mr-auto">
            <div class="border text-center">
                {% image page.image max-370x270 as temp_image %}
              <img src="{{ temp_image.url }}" alt="Image" class="img-fluid p-5">
            div>
          div>
          <div class="col-md-6">
            <h2 class="text-black">{{ page.title }}h2>
            <p>{{page.short_description}}p>


            <p><del>$95.00del>  <strong class="text-primary h4">Ksh. {{ page.price }}strong>p>

            <p>
                {% for f in custom_fields %}
                    {% if f.options_array|length > 0 %}
                        <div class="form-group">
                            <label class="form-label" for="{{ f.name|lower }}">
                                {{ f.name }}:
                            label>
                            <select class="form-select custom-field-select" id="{{ f.name|lower }}" data-field="{{ forloop.counter }}">
                                {% for opt in f.options_array %}
                                    <option>
                                        {{ opt }}
                                    option>
                                {% endfor %}
                            select>
                        div>
                    {% endif %}
                {% endfor %}
            p>


            <button class="snipcart-add-item btn btn-primary mt-6 py-2 px-4 bg-gradient-to-r from-green-400 to-blue-500 hover:from-pink-500 hover:to-yellow-500 text-white font-bold rounded-full shadow-offset hover:shadow-lg transition duration-300"
                data-item-name="{{ page.title }}"
                    data-item-id="{{ page.sku }}"
                    data-item-url="{{ page.get_full_url }}"
                    data-item-price="{{ page.price }}"
                    data-item-description="{{ page.short_description}}"
                    data-item-image="{{ temp_image.url }}"
                    {% for f in custom_fields %}
                        data-item-custom{{forloop.counter}}-name="{{f.name}}"
                        data-item-custom{{forloop.counter}}-options="{{f.options}}"
                    {% endfor %}>Add to cart
            button>


            <div class="mt-5">
              <ul class="nav nav-pills mb-3 custom-pill" id="pills-tab" role="tablist">
                <li class="nav-item">
                  <a class="nav-link active" id="pills-home-tab" data-toggle="pill" href="#pills-home" role="tab"
                    aria-controls="pills-home" aria-selected="true">Ordering Informationa>
                li>
                <li class="nav-item">
                  <a class="nav-link" id="pills-profile-tab" data-toggle="pill" href="#pills-profile" role="tab"
                    aria-controls="pills-profile" aria-selected="false">Specificationsa>
                li>

              ul>
              <div class="tab-content" id="pills-tabContent">
                <div class="tab-pane fade show active" id="pills-home" role="tabpanel" aria-labelledby="pills-home-tab">
                  <table class="table custom-table">
                    <thead>
                      <th>Materialth>
                      <th>Descriptionth>
                      <th>Packagingth>
                    thead>
                    <tbody>
                      <tr>
                        <th scope="row">OTC022401th>
                        <td>Pain Management: Acetaminophen PM Extra-Strength Caplets, 500 mg, 100/Bottletd>
                        <td>1 BTtd>
                      tr>
                      <tr>
                        <th scope="row">OTC022401th>
                        <td>Pain Management: Acetaminophen PM Extra-Strength Caplets, 500 mg, 100/Bottletd>
                        <td>144/CStd>
                      tr>
                      <tr>
                        <th scope="row">OTC022401th>
                        <td>Pain Management: Acetaminophen PM Extra-Strength Caplets, 500 mg, 100/Bottletd>
                        <td>1 EAtd>
                      tr>

                    tbody>
                  table>
                div>
                <div class="tab-pane fade" id="pills-profile" role="tabpanel" aria-labelledby="pills-profile-tab">

                  <table class="table custom-table">

                    <tbody>
                      <tr>
                        <td>HPIS CODEtd>
                        <td class="bg-light">999_200_40_0td>
                      tr>
                      <tr>
                        <td>HEALTHCARE PROVIDERS ONLYtd>
                        <td class="bg-light">Notd>
                      tr>
                      <tr>
                        <td>LATEX FREEtd>
                        <td class="bg-light">Yes, Notd>
                      tr>
                      <tr>
                        <td>MEDICATION ROUTEtd>
                        <td class="bg-light">Topicaltd>
                      tr>
                    tbody>
                  table>

                div>

              div>
            div>


          div>
        div>
      div>
    div>

    <div class="site-section bg-secondary bg-image" style="background-image: url('images/bg_2.jpg');">
      <div class="container">
        <div class="row align-items-stretch">
          <div class="col-lg-6 mb-5 mb-lg-0">
            <a href="#" class="banner-1 h-100 d-flex" style="background-image: url('images/bg_1.jpg');">
              <div class="banner-1-inner align-self-center">
                <h2>Tafari Pharmacy Productsh2>
                <p>Lorem, ipsum dolor sit amet consectetur adipisicing elit. Molestiae ex ad minus rem odio voluptatem.
                p>
              div>
            a>
          div>
          <div class="col-lg-6 mb-5 mb-lg-0">
            <a href="#" class="banner-1 h-100 d-flex" style="background-image: url('images/bg_2.jpg');">
              <div class="banner-1-inner ml-auto  align-self-center">
                <h2>Rated by Expertsh2>
                <p>Lorem, ipsum dolor sit amet consectetur adipisicing elit. Molestiae ex ad minus rem odio voluptatem.
                p>
              div>
            a>
          div>
        div>
      div>
    div>


    <footer class="site-footer">
      <div class="container">
        <div class="row">
          <div class="col-md-6 col-lg-3 mb-4 mb-lg-0">

            <div class="block-7">
              <h3 class="footer-heading mb-4">About Ush3>
              <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Eius quae reiciendis distinctio voluptates
                sed dolorum excepturi iure eaque, aut unde.p>
            div>

          div>
          <div class="col-lg-3 mx-auto mb-5 mb-lg-0">
            <h3 class="footer-heading mb-4">Quick Linksh3>
            <ul class="list-unstyled">
              <li><a href="#">Supplementsa>li>
              <li><a href="#">Vitaminsa>li>
              <li><a href="#">Diet & Nutritiona>li>
              <li><a href="#">Tea & Coffeea>li>
            ul>
          div>

          <div class="col-md-6 col-lg-3">
            <div class="block-5 mb-5">
              <h3 class="footer-heading mb-4">Contact Infoh3>
              <ul class="list-unstyled">
                <li class="address">203 Fake St. Mountain View, San Francisco, California, USAli>
                <li class="phone"><a href="tel://23923929210">+2 392 3929 210a>li>
                <li class="email">emailaddress@domain.comli>
              ul>
            div>


          div>
        div>
        <div class="row pt-5 mt-5 text-center">
          <div class="col-md-12">
            <p>
              
              Copyright ©
              <script>document.write(new Date().getFullYear());script> All rights reserved | This template is made
              with <i class="icon-heart" aria-hidden="true">i> by <a href="https://colorlib.com" target="_blank"
                class="text-primary">Colorliba>
              
            p>
          div>

        div>
      div>
    footer>
  div>

  <script src="{% static 'js/jquery-3.3.1.min.js' %}">script>
  <script src="{% static 'js/jquery-ui.js' %}">script>
  <script src="{% static 'js/popper.min.js' %}">script>
  <script src="{% static 'js/bootstrap.min.js' %}">script>
  <script src="{% static 'js/owl.carousel.min.js' %}">script>
  <script src="{% static 'js/jquery.magnific-popup.min.js' %}">script>
  <script src="{% static 'js/aos.js' %}">script>

  <script src="{% static 'js/main.js' %}">script>
  <script>
    document.addEventListener('DOMContentLoaded', function() {
        document.querySelector('.custom-field-select').onchange = function(event) {
            if (event.target.dataset.field) {
                document.querySelector('.snipcart-add-item')
                    .dataset['itemCustom' + event.target.dataset.field + 'Value'] = event.target.value;
            }
        };
    },false);
script>
body>

html>

The javascript included above is responsible for updating the Snipcart buy button when a custom field selection is made on the page. This code updates the button data attributes when the select value changes.

Live demo

See the live demo

Final thoughts about the project

Wagtail is simple, intuitive and minimal. I think wagtail is a great headless CMS, and I have enjoyed my little venture with Django, Wagtail CMS, and Bootstrap. LEt me know what you think about Wagtail in the comment section!

Personality Prediction Using K-Means Clustering Algorithm and Django Rest Framework (Article Partly Written By GPT3)

Paul Kiragu — Wed, 26 Jan 2022 11:28:21 GMT

Introduction

Artificial Intelligence is revolutionizing the world today as we know it through technical transformations. Machine learning applications are applied in our day-to-day lives, and one of the incredible applications of machine learning is to classify individuals based on their personality traits. Each person on this planet is unique and carries a unique personality. The availability of a high-dimensional and large amount of data has paved the way for increasing marketing campaigns' effectiveness by targeting specific people. Such personality-based communications are highly effective in increasing the popularity and attractiveness of products and services.

In this article, we build a model to predict personality based on 50 questions and deploy the model using Django Rest Framework.

I will also be sharing a snapshot of my code on how I was able to use GPT3 to help me generate content in this article. Every text below, except 'the steps to implement' section, was generated by GPT3, a natural-language processing system that can generate tweets, pens poetry, summarizes emails, answers trivia questions, translates languages and even write its own computer programs.

K-means Clustering Algorithm.

The K-Means clustering algorithm is an algorithm that is used in unsupervised machine learning. To put it simply, it is a clustering algorithm where it groups the data into clusters and then assigns a label to each of the clusters. The K-Means algorithm is an iterative algorithm, it means that it repeats the following steps in a loop until the algorithm is satisfied.

Assign the input data to K clusters (there are K clusters, the algorithm sorts the data into K clusters)
Assign each input data to the closest cluster (the algorithm calculates the distance between the data and each of the clusters and then assigns each data to the closest cluster)
Update the cluster centroids (this is the average of all the data in the cluster)

Applications of K-means Algorithm.

The k-means clustering algorithm is used in a variety of applications. It is used in marketing, advertisement, market segmentation, and customer segmentation. It is also used in various scientific fields, like in determining someones personality.nIn determining someones personality.

Five Personality Traits (OCEAN)

The big five personality traits are:

Openness to experience
Conscientiousness
Extraversion
Agreeableness
Neuroticism.

These are the five traits that human behavior is divided into. People can score between 1 and 5 in each of the five traits. For example, let's say that someone scores 3 in openness to experience. This means that he/she is not very open to new experiences, he/she is a person that is very comfortable with their own ways and he/she wants to stick to it. He/she is not very open to new things and he/she tries to avoid it. He/she is a person that doesn't like to try new things. For example, in a party, he/she is not very social and he/she doesn't like to talk a lot. He/she is a person that speaks little and he/she is a person that is not very sociable. In a party, he/she likes to stay at the corner and he/she won't go talk to other people. He/she will just stay and look at other people in the party. This is his/her personality.

Now that we have a basic understanding of the k-means algorithm and the big five personality traits, let us dive deep into building the Big Five personality traits.

Steps to implement.

Let us start by building and saving our model that will be later used to make predictions for our API.

Dataset.

Our dataset consists of 1,015,342 questionnaire answers collected online by Open Psychometrics. Let's look at how the dataset actually appears. The dataset and entire code for this blog can be found on my Github Repo

We first import the necessary dependencies. If you do not have the libraries installed, kindly do so before proceeding.

import numpy as np 
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os
from sklearn.cluster import KMeans
from yellowbrick.cluster import KElbowVisualizer
from sklearn.preprocessing import MinMaxScaler
from sklearn.decomposition import PCA
import joblib

Next, we load the dataset using pandas and then we display the number of participants:

data_raw = pd.read_csv('data-final.csv', sep='\t')
data = data_raw.copy()
pd.options.display.max_columns = 150

data.drop(data.columns[50:107], axis=1, inplace=True)
data.drop(data.columns[51:], axis=1, inplace=True)

print('Number of participants: ', len(data))
data.head()

Number of participants:  1015341

Exploration of Dataset.

Let us begin by checking for missing values and removing the missing values, like so:

print('Missing value? ', data.isnull().values.any())
print('How many? ', data.isnull().values.sum())
data.dropna(inplace=True)
print('Number of participants after eliminating missing values: ', len(data))

Is there any missing value?  True
How many missing values?  89227
Number of participants after eliminating missing values:  1013481

Let us now look at the participant distribution per nationality, by entering the code:

# Participants' nationality distriution
countries = pd.DataFrame(data['country'].value_counts())
countries_5000 = countries[countries['country'] >= 5000]
plt.figure(figsize=(15,5))
sns.barplot(data=countries_5000, x=countries_5000.index, y='country')
plt.title('Countries With More Than 5000 Participants')
plt.ylabel('Participants');

Let us now visualize the question and answer distribution of the questionnaires, like so:

# Defining a function to visualize the questions and answers distribution
def vis_questions(groupname, questions, color):
    plt.figure(figsize=(40,60))
    for i in range(1, 11):
        plt.subplot(10,5,i)
        plt.hist(data[groupname[i-1]], bins=14, color= color, alpha=.5)
        plt.title(questions[groupname[i-1]], fontsize=18)

print('Q&As Related to Extroversion Personality')
vis_questions(EXT, ext_questions, 'orange')

print('Q&As Related to Neuroticism Personality')
vis_questions(EST, est_questions, 'pink')

print('Q&As Related to Agreeable Personality')
vis_questions(AGR, agr_questions, 'red')

print('Q&As Related to Conscientious Personality')
vis_questions(CSN, csn_questions, 'purple')

print('Q&As Related to Open Personality')
vis_questions(OPN, opn_questions, 'blue')

Building the Model

We already know the number of clusters that will be present in our model, that is 5, the big five of personality traits. Let us see how we can get this value using code. For ease of calculation, we shall scale all the values between 0-1 and use only a sample of 5000, like so:

df = data.drop('country', axis=1)
columns = list(df.columns)

scaler = MinMaxScaler(feature_range=(0,1))
df = scaler.fit_transform(df)
df = pd.DataFrame(df, columns=columns)
df_sample = df[:5000]

Let us now visualize our elbow curve. In cluster analysis, the elbow method is a heuristic used in determining the number of clusters in a data set. The method consists of plotting the explained variation as a function of the number of clusters, and picking the elbow of the curve as the number of clusters to use.

kmeans = KMeans()
visualizer = KElbowVisualizer(kmeans, k=(2,15))
visualizer.fit(df_sample)
visualizer.poof()

As you can see 5 clusters looks optimum for the data set and we already know this researh is to identify 5 different personalities.

Clustering Participants into 5 Personality Groups

To do this, we use the unscaled data but without the country column.

df_model = data.drop('country', axis=1)

# I define 5 clusters and fit my model
kmeans = KMeans(n_clusters=5)
k_fit = kmeans.fit(df_model)
## SAVE KFIT MODEL
joblib_file = "BigFivePersonalityTestModel.joblib"
joblib.dump(k_fit, joblib_file)

# Predicting the Clusters
pd.options.display.max_columns = 10
predictions = k_fit.labels_
df_model['Clusters'] = predictions
df_model.head()


EXT1    EXT2    EXT3    EXT4    EXT5    ...    OPN7    OPN8    OPN9    OPN10    Clusters
0    4.0    1.0    5.0    2.0    5.0    ...    5.0    3.0    4.0    5.0    3
1    3.0    5.0    3.0    4.0    3.0    ...    4.0    2.0    5.0    3.0    2
2    2.0    3.0    4.0    4.0    3.0    ...    5.0    3.0    4.0    4.0    2
3    2.0    2.0    2.0    3.0    4.0    ...    4.0    4.0    3.0    3.0    1
4    3.0    3.0    3.0    3.0    5.0    ...    5.0    3.0    5.0    5.0    3

Analysing the Model and Predictions

Let us see how many individuals we have for each cluster, like so:

df_model.Clusters.value_counts()

4    227063
2    212816
3    210075
0    200226
1    163301
Name: Clusters, dtype: int64

Let's group the results acording to clusters. That way we can investigate the average answer to the each question for each cluster.

That way we can have an intuition about how our model classifies people.

pd.options.display.max_columns = 150
df_model.groupby('Clusters').mean()

Let's now sum up the each question groups (EXT, EST ..) and see if we can see a pattern.

# Summing up the different questions groups
col_list = list(df_model)
ext = col_list[0:10]
est = col_list[10:20]
agr = col_list[20:30]
csn = col_list[30:40]
opn = col_list[40:50]

data_sums = pd.DataFrame()
data_sums['extroversion'] = df_model[ext].sum(axis=1)/10
data_sums['neurotic'] = df_model[est].sum(axis=1)/10
data_sums['agreeable'] = df_model[agr].sum(axis=1)/10
data_sums['conscientious'] = df_model[csn].sum(axis=1)/10
data_sums['open'] = df_model[opn].sum(axis=1)/10
data_sums['clusters'] = predictions
data_sums.groupby('clusters').mean()


extroversion    neurotic    agreeable    conscientious    open
clusters                    
0    2.965969    3.645931    3.148628    3.173210    3.245529
1    2.909467    2.525743    2.851802    2.914458    3.120373
2    3.051889    2.984940    3.187544    3.159140    3.243641
3    3.085431    2.423577    3.209064    3.106899    3.327173
4    3.072319    3.426610    3.300147    3.211454    3.352370

Let us now visualize the mean for each of our 5 personality clusters:

# Visualizing the means for each cluster
dataclusters = data_sums.groupby('clusters').mean()
plt.figure(figsize=(22,3))
for i in range(0, 5):
    plt.subplot(1,5,i+1)
    plt.bar(dataclusters.columns, dataclusters.iloc[:, i], color='green', alpha=0.2)
    plt.plot(dataclusters.columns, dataclusters.iloc[:, i], color='red')
    plt.title('Cluster ' + str(i))
    plt.xticks(rotation=45)
    plt.ylim(0,4);

Visualizing the Cluster Predictions

pca = PCA(n_components=2)
pca_fit = pca.fit_transform(df_model)

df_pca = pd.DataFrame(data=pca_fit, columns=['PCA1', 'PCA2'])
df_pca['Clusters'] = predictions
df_pca.head()
plt.figure(figsize=(10,10))
sns.scatterplot(data=df_pca, x='PCA1', y='PCA2', hue='Clusters', palette='Set2', alpha=0.8)
plt.title('Personality Clusters after PCA');

Predict Personality

I answered the questions in an Microsoft Excel spread sheet. Then I added that data into this notebook and put my answers to the model to see in which category I will be.

my_data = pd.read_excel('my_personality_test.xlsx')
my_personality = k_fit.predict(my_data)
print('My Personality Cluster: ', my_personality)

My Personality Cluster:  [2]

Visualizing my personality cluster

col_list = list(my_data)
ext = col_list[0:10]
est = col_list[10:20]
agr = col_list[20:30]
csn = col_list[30:40]
opn = col_list[40:50]

my_sums = pd.DataFrame()
my_sums['extroversion'] = my_data[ext].sum(axis=1)/10
my_sums['neurotic'] = my_data[est].sum(axis=1)/10
my_sums['agreeable'] = my_data[agr].sum(axis=1)/10
my_sums['conscientious'] = my_data[csn].sum(axis=1)/10
my_sums['open'] = my_data[opn].sum(axis=1)/10
my_sums['cluster'] = my_personality
print('Sum of my question groups')
my_sums
my_sum = my_sums.drop('cluster', axis=1)
plt.bar(my_sum.columns, my_sum.iloc[0,:], color='green', alpha=0.2)
plt.plot(my_sum.columns, my_sum.iloc[0,:], color='red')
plt.title('Cluster 2')
plt.xticks(rotation=45)
plt.ylim(0,4);

Now that our model works, let us proceed by turning our model into a Restful API

Turning the Model into an RESTFUL API

Following Python best practices, we will create a virtual environment for our project, and install the required packages.

First, create the project directory.

$ mkdir djangoapp
$ cd djangoapp

Now, create a virtual environment and install the required packages.

For macOS and Unix systems:

$ python3 -m venv myenv
$ source myenv/bin/activate
(myenv) $ pip install django requests numpy joblib scikit-learn xlsxwriter openpyxl

For Windows:

$ python3 -m venv myenv
$ myenv\Scripts\activate
(myenv) $ pip install django requests numpy joblib scikit-learn xlsxwriter openpyxl

Setting Up Your Django Application

First, navigate to the directory djangoapp we created and establish a Django project.

(myenv) $ django-admin startproject mainapp

This will auto-generate some files for your project skeleton:

mainapp/
    manage.py
    mainapp/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Now, navigate to the directory you just created (make sure you are in the same directory as manage.py) and create your app directory.

(myenv) $ python manage.py startapp monitor

This will create the following:

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
        __init__.py
    models.py
    tests.py
    views.py

On the mainapp/settings.py file, look for the following line and add the app we just created above.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'monitor', #new line
]

Ensure you are in the monitor directory then create a new directory called templates then a new file called urls.py. Your directory structure of monitor application should look like this

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
    templates/
        __init__.py
    models.py
    tests.py
    urls.py
    views.py

Ensure your mainapp/urls.py file, add our monitor app URL to include the URLs we shall create next on the monitor app:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    #path('admin/', admin.site.urls),
    path('', include('monitor.urls')),#monitor app url
]

Now, on the monitor/urls.py file, add our website there:

from django.urls import path
from .views import *

urlpatterns = [
    path('persona', PersonalityPrediction.as_view(), name='personality'),
]

Let’s create another directory to store our machine learning model. I’ll also add the dataset to the project for those who want to achieve the whole dataset. (It is not compulsory to create a data folder.) Be sure to move the vectorizer file and the joblib file we created earlier to ml/model folder

(venv)$ mkdir ml
(venv)$ mkdir ml/models
(venv)$ mkdir ml/data

We also need to tell Django where our machine learning model is located. Add these lines to settings.py file:

import os
MODELS = os.path.join(BASE_DIR, 'ml/models')

Load Model and Vectorizer through apps.py

Load the model we created and saved in apps.py so that when the application starts, the trained model is loaded only once. Otherwise, the trained model is loaded each time an endpoint is called, and then the response time will be slower.

Let’s update apps.py

import os
import joblib
from django.apps import AppConfig
from django.conf import settings


class ApiConfig(AppConfig):
    name = 'api'
    MODEL_FILE = os.path.join(settings.MODELS, "BigFivePersonalityTestModel.joblib")
    model = joblib.load(MODEL_FILE)

Edit views.py

The views will be mainly responsible for:

Process incoming POST requests.

from django.shortcuts import render
from rest_framework.views import APIView
from rest_framework.response import Response
import requests
from .apps import *
import pandas as pd
import xlsxwriter

#kmeans personality prediction view
class PersonalityPrediction(APIView):
    def post(self, request):
        #get user input
        EXT1 = request.data.get('EXT1')
        EXT2 = request.data.get('EXT2')
        EXT3 = request.data.get('EXT3')
        EXT4 = request.data.get('EXT4')
        EXT5 = request.data.get('EXT5')
        EXT6 = request.data.get('EXT6')
        EXT7 = request.data.get('EXT7')
        EXT8 = request.data.get('EXT8')
        EXT9 = request.data.get('EXT9')
        EXT10 = request.data.get('EXT10')

        EST1 = request.data.get('EST1')
        EST2 = request.data.get('EST2')
        EST3 = request.data.get('EST3')
        EST4 = request.data.get('EST4')
        EST5 = request.data.get('EST5')
        EST6 = request.data.get('EST6')
        EST7 = request.data.get('EST7')
        EST8 = request.data.get('EST8')
        EST9 = request.data.get('EST9')
        EST10 = request.data.get('EST10')

        AGR1 = request.data.get('AGR1')
        AGR2 = request.data.get('AGR2')
        AGR3 = request.data.get('AGR3')
        AGR4 = request.data.get('AGR4')
        AGR5 = request.data.get('AGR5')
        AGR6 = request.data.get('AGR6')
        AGR7 = request.data.get('AGR7')
        AGR8 = request.data.get('AGR8')
        AGR9 = request.data.get('AGR9')
        AGR10 = request.data.get('AGR10')

        CSN1 = request.data.get('CSN1')
        CSN2 = request.data.get('CSN2')
        CSN3 = request.data.get('CSN3')
        CSN4 = request.data.get('CSN4')
        CSN5 = request.data.get('CSN5')
        CSN6 = request.data.get('CSN6')
        CSN7 = request.data.get('CSN7')
        CSN8 = request.data.get('CSN8')
        CSN9 = request.data.get('CSN9')
        CSN10 = request.data.get('CSN10')

        OPN1 = request.data.get('OPN1')
        OPN2 = request.data.get('OPN2')
        OPN3 = request.data.get('OPN3')
        OPN4 = request.data.get('OPN4')
        OPN5 = request.data.get('OPN5')
        OPN6 = request.data.get('OPN6')
        OPN7 = request.data.get('OPN7')
        OPN8 = request.data.get('OPN8')
        OPN9 = request.data.get('OPN9')
        OPN10 = request.data.get('OPN10')
        #load model
        kmeansModel = PersonalityTestConfig.model
        # Predicting the Clusters
        pd.options.display.max_columns = 10
        predictions = kmeansModel.labels_

        workbook = xlsxwriter.Workbook('hello.xlsx')

        worksheet = workbook.add_worksheet()

        worksheet.write('A1', 'EXT1')
        worksheet.write('A2', (EXT1))

        worksheet.write('B1', 'EXT2')
        worksheet.write('B2', (EXT2))

        worksheet.write('C1', 'EXT3')
        worksheet.write('C2', (EXT3))

        worksheet.write('D1', 'EXT4')
        worksheet.write('D2', (EXT4))

        worksheet.write('E1', 'EXT5')
        worksheet.write('E2', (EXT5))

        worksheet.write('F1', 'EXT6')
        worksheet.write('F2', (EXT6))

        worksheet.write('G1', 'EXT7')
        worksheet.write('G2', (EXT7))

        worksheet.write('H1', 'EXT8')
        worksheet.write('H2', (EXT8))

        worksheet.write('I1', 'EXT9')
        worksheet.write('I2', (EXT9))

        worksheet.write('J1', 'EXT10')
        worksheet.write('J2', (EXT10))

        worksheet.write('K1', 'EST1')
        worksheet.write('K2', (EST1))

        worksheet.write('L1', 'EST2')
        worksheet.write('L2', (EST2))

        worksheet.write('M1', 'EST3')
        worksheet.write('M2', (EST3))

        worksheet.write('N1', 'EST4')
        worksheet.write('N2', (EST4))

        worksheet.write('O1', 'EST5')
        worksheet.write('O2', (EST5))

        worksheet.write('P1', 'EST6')
        worksheet.write('P2', (EST6))

        worksheet.write('Q1', 'EST7')
        worksheet.write('Q2', (EST7))

        worksheet.write('R1', 'EST8')
        worksheet.write('R2', (EST8))

        worksheet.write('S1', 'EST9')
        worksheet.write('S2', (EST9))

        worksheet.write('T1', 'EST10')
        worksheet.write('T2', (EST10))

        worksheet.write('U1', 'AGR1')
        worksheet.write('U2', (AGR1))

        worksheet.write('V1', 'AGR2')
        worksheet.write('V2', (AGR2))

        worksheet.write('W1', 'AGR3')
        worksheet.write('W2', (AGR3))

        worksheet.write('X1', 'AGR4')
        worksheet.write('X2', (AGR4))

        worksheet.write('Y1', 'AGR5')
        worksheet.write('Y2', (AGR5))

        worksheet.write('Z1', 'AGR6')
        worksheet.write('Z2', (AGR6))

        worksheet.write('AA1', 'AGR7')
        worksheet.write('AA2', (AGR7))

        worksheet.write('AB1', 'AGR8')
        worksheet.write('AB2', (AGR8))

        worksheet.write('AC1', 'AGR9')
        worksheet.write('AC2', (AGR9))

        worksheet.write('AD1', 'AGR10')
        worksheet.write('AD2', (AGR10))


        worksheet.write('AE1', 'CSN1')
        worksheet.write('AE2', (CSN1))

        worksheet.write('AF1', 'CSN2')
        worksheet.write('AF2', (CSN2))

        worksheet.write('AG1', 'CSN3')
        worksheet.write('AG2', (CSN3))

        worksheet.write('AH1', 'CSN4')
        worksheet.write('AH2', (CSN4))

        worksheet.write('AI1', 'CSN5')
        worksheet.write('AI2', (CSN5))

        worksheet.write('AJ1', 'CSN6')
        worksheet.write('AJ2', (CSN6))

        worksheet.write('AK1', 'CSN7')
        worksheet.write('AK2', (CSN7))

        worksheet.write('AL1', 'CSN8')
        worksheet.write('AL2', (CSN8))

        worksheet.write('AM1', 'CSN9')
        worksheet.write('AM2', (CSN9))

        worksheet.write('AN1', 'CSN10')
        worksheet.write('AN2', (CSN10))


        worksheet.write('AO1', 'OPN1')
        worksheet.write('AO2', (OPN1))

        worksheet.write('AP1', 'OPN2')
        worksheet.write('AP2', (OPN2))

        worksheet.write('AQ1', 'OPN3')
        worksheet.write('AQ2', (OPN3))

        worksheet.write('AR1', 'OPN4')
        worksheet.write('AR2', (OPN4))

        worksheet.write('AS1', 'OPN5')
        worksheet.write('AS2', (OPN5))

        worksheet.write('AT1', 'OPN6')
        worksheet.write('AT2', (OPN6))

        worksheet.write('AU1', 'OPN7')
        worksheet.write('AU2', (OPN7))

        worksheet.write('AV1', 'OPN8')
        worksheet.write('AV2', (OPN8))

        worksheet.write('AW1', 'OPN9')
        worksheet.write('AW2', (OPN9))

        worksheet.write('AX1', 'OPN10')
        worksheet.write('AX2', (OPN10))


        # Finally, close the Excel file
        # via the close() method.
        workbook.close()

        my_data = pd.read_excel('hello.xlsx', engine='openpyxl')

        my_personality = kmeansModel.predict(my_data)
        print('My Personality Cluster: ', my_personality)

        # Summing up the my question groups
        col_list = list(my_data)
        ext = col_list[0:10]
        est = col_list[10:20]
        agr = col_list[20:30]
        csn = col_list[30:40]
        opn = col_list[40:50]

        my_sums = pd.DataFrame()
        my_sums['extroversion'] = my_data[ext].sum(axis=1)/10
        my_sums['neurotic'] = my_data[est].sum(axis=1)/10
        my_sums['agreeable'] = my_data[agr].sum(axis=1)/10
        my_sums['conscientious'] = my_data[csn].sum(axis=1)/10
        my_sums['open'] = my_data[opn].sum(axis=1)/10
        my_sums['cluster'] = my_personality
        print('Sum of my question groups')
        print(my_sums)

        response_dict = {"Prediction": my_sums}
        print(response_dict)
        return Response(response_dict, status=200)

The code above starts by getting the data appended to the request body. The code then creates a new excel file programmatically which is used in the prediction and returns the cluster as a response.

Testing our model.

To test the system, make the necessary migrations and run the django server. Open POSTMAN and make a POST request to our server like so, with the answers to the questionnaire appended to the body of the request. We should get a sample response showing us our personality cluster and our scores. To test my live system, make a POST request to the URL below:

https://alienx.tech/api/v1/persona

The code below is what i used to prompt GPT3 to help me generate the content in this blog:

import openai
openai.api_key = "VISIT OPENAI TO GET YOUR KEY"
response = openai.Completion.create(
  engine="davinci",
  prompt="The Pixel District Janury 16, 2022n Title: Personality Segmentation Using K-Means Clustering Algorithm and Django Rest Framework!n tags: machine-learning, kmeans, gpt3, kmeans code sample Summary:  I am sharing my exprience in implementing kmeans clustering algorithmn in determining someones personality. I am explaining why kmeans clustering algorithms is, how it works configuration. I am explaining what the big five personality traits are. I am explaining why I think kmeans algorithm is the best to use in finding someones personality based on the  big five personality traits. I am also adding various example codes of the kmeans clustering to find the big five personality traits.n Full text: ",
  temperature=0.7,
  max_tokens=1655,
  top_p=1,
  frequency_penalty=0,
  presence_penalty=0
)
print(response)

That's it for today. Thanks for staying tuned in!

How to Restore Dislikes on YouTube with Django Rest Framework and YoutubeV3 API.

Paul Kiragu — Wed, 01 Dec 2021 15:23:34 GMT

UPDATE

Youtube has taken down the view count API and this functionality is no longer functional

Overview:

Earlier this month, video sharing service YouTube, which is now the second-most-used platform online, announced that while the "dislike" button would remain, the actual dislike count would be made private.

"To ensure that YouTube promotes respectful interactions between viewers and creators, we introduced several features and policies to improve their experience," the company announced via a blog post. "And earlier this year, we experimented with the dislike button to see whether or not changes could help better protect our creators from harassment, and reduce dislike attacks — where people work to drive up the number of dislikes on a creator’s videos."

While the removal of public dislikes could help the creators, there is the argument this is also about helping YouTube by getting more eyeballs on videos – even ones that could disappoint some viewers. The lack of "dislikes" means there is no quick public review of the user-generated content.

Introduction

In this article, we will see how we can display the dislike count of any youtube video using YoutubeV3 API and Django Rest Framework.

To test my live system, head over to a youtube video of your choice on the web:

Replace "www.youtube.com":

With "alienx.tech":

Be sure not to change the video-ID and ensure you remove the "www" otherwise you will get a privacy error, then hit enter.

If successful, your browser will redirect to the page below which displays both the like and dislike count:

Before we dive into building our microservice, let us start by understanding how I was able to achieve this.

How it Works

It is worth stating that I got this idea from y2mate which is a free youtube downloader website that works by inserting "pp" after the word "youtube" in the link, like so:

The first thing I needed to do was to create an API route to be like youtube is on my Django backend. I also needed the youtube API for collecting the data about the video at play. Surprisingly, youtube has not made that functionality private which is fair enough.

I did not find Youtubes' API docs to be user friendly and so I did some digging where I came across this god-sent freemium app on RapidAPI that enables you to use the youtube V3 API without having an API key from Google developers account! The app is very well maintained and has a low ping. PS: I was able to tell it's an awesome API by just looking at the ranking score it got on rapid API so negative feedback can actually help save you money;). Imagine subscribing to a paid API that turns out to be very slow yet you could have looked at the like to dislike ratio.

On my Django Rest backend side, I needed prometheus,(the name of my API server), to make a request to youtubeV3 API, then return the response to the user as a template, not json or any other format. Django being the most powerful web development framework, in my opinion, has an inbuilt function for doing this called Renderers. The rendering process works by taking the intermediate representation of the template and context and turning it into the final byte stream that can be served to the client.

Building our API

For this tutorial, we shall be using python3.5+, Django Rest Web Framework, and Youtube's v3 API for pulling youtube video data including the dislike count.

Following Python best practices, we will create a virtual environment for our project, and install the required packages.

First, create the project directory.

$ mkdir djangoapp
$ cd djangoapp

Now, create a virtual environment and install the required packages.

For macOS and Unix systems:

$ python3 -m venv myenv
$ source myenv/bin/activate
(myenv) $ pip3 install django djangorestframework requests

For Windows:

$ python3 -m venv myenv
$ myenv\Scripts\activate
(myenv) $ pip3 install django djangorestframework requests

Youtube V3 API

As mentioned earlier, I will not be accessing youtube's API using my own access token, I will be using a third-party app found on rapid API which is one of the largest API marketplaces on the web. Create an account with rapid API to get the freemium youtubeV3API

Setting Up Your Django Application

First, navigate to the directory Django app we created and establish a Django project.

(myenv) $ django-admin startproject mainapp

This will auto-generate some files for your project skeleton:

mainapp/
    manage.py
    mainapp/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Now, navigate to the directory you just created (make sure you are in the same directory as manage.py) and create your app directory.

(myenv) $ python manage.py startapp exam

This will create the following:

exam/
    __init__.py
    admin.py
    apps.py
    migrations/
        __init__.py
    models.py
    tests.py
    views.py

On the mainapp/settings.py file, look for the following line and add the app we just created above.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.humanize',#new line, explanation later
    'rest_framework',#new line
    'exam', #new line
]

Ensure you are in the exam directory then create a new directory called templates and a new file called urls.py. Your directory structure of exam application should look like this

exam/
    __init__.py
    admin.py
    apps.py
    migrations/
    templates/
        __init__.py
    models.py
    tests.py
    urls.py
    views.py

Ensure your mainapp/urls.py file, add our exam app URL to include the URLs we shall create next on the exam app:

"""mainapp URL Configuration

The `urlpatterns` list routes URLs to views. For more information please see:
    https://docs.djangoproject.com/en/3.2/topics/http/urls/
Examples:
Function views
    1. Add an import:  from my_app import views
    2. Add a URL to urlpatterns:  path('', views.home, name='home')
Class-based views
    1. Add an import:  from other_app.views import Home
    2. Add a URL to urlpatterns:  path('', Home.as_view(), name='home')
Including another URLconf
    1. Import the include() function: from django.urls import include, path
    2. Add a URL to urlpatterns:  path('blog/', include('blog.urls'))
"""
from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path(' ', include('exam.urls')),#exam app url
]

Now, on the exam/urls.py file, add our website there:

from django.urls import path, include
from . import views

urlpatterns = [
    path('watch', YoutubeStats.as_view(), name="youtubestats"),
]

On the exam/views.py, add the following lines of code:

from django.shortcuts import render, redirect
import requests
from django.http import HttpResponse, HttpResponseRedirect
from rest_framework.response import Response
from rest_framework.views import APIView
from rest_framework.renderers import TemplateHTMLRenderer

class YoutubeStats(APIView):
      renderer_classes = [TemplateHTMLRenderer]
      template_name = 'youtube.html'

      def get(self, request):
          toScan = request.GET.get('v') #get users request(video URL in this case)
          #youtubev3 api
          parsed = str(toScan) #convert to string
          url = 'https://youtube-v31.p.rapidapi.com/videos'
          querystring = {"part":"contentDetails,snippet,statistics","id":parsed} 
          headers = {'x-rapidapi-host': "youtube-v31.p.rapidapi.com",'x-rapidapi-key': "GETYOURKEY"}
          response = requests.request("GET", url, headers=headers, params=querystring)
          DataResponse = response.json() 
          items = DataResponse.get('items')
          for data in items:
              chanid = data.get('snippet')
              channelid = chanid['channelId']
          #context = {'items':items, 'videoid':parsed}
          url2 = "https://youtube-v31.p.rapidapi.com/channels"
          querystring2 = {"part":"snippet,statistics","id":channelid}
          response2 = requests.request("GET", url2, headers=headers, params=querystring2)
          DataResponse2 = response2.json()
          items2 = DataResponse2.get('items')
          context = {'items':items, 'videoid':parsed, 'items2':items2,}
          return Response(context)

Explanation of views.py

In the code above, we start by getting initiating the template class and template name of our API. We then get the request of the user that was appended to the video to be analyzed ('v'). We then proceed by converting the requested videoID to a string and then making a request to the youtube API.

The sample response from youtube API looks like so:

{
   "kind":"youtube#videoListResponse",
   "items":[
      {
         "kind":"youtube#video",
         "id":"frn3WT15Dy8",
         "snippet":{
            "publishedAt":"2021-11-23T01:30:12Z",
            "channelId":"UCPWXiRWZ29zrxPFIQT7eHSA",
            "title":"'This Is Now Becoming A Pattern': Jordan Slams YouTube Covid-19 'Censorship'",
            "description":"During a Wednesday hearing on COVID-19 misinformation, Rep. Jim Jordan (R-OH) criticized YouTube for removing a video containing 'misinformation.' The Ohio Republican said the social media company should not be censoring doctors over 'science.'",
            "thumbnails":{
               "default":{
                  "url":"https://i.ytimg.com/vi/frn3WT15Dy8/default.jpg",
                  "width":120,
                  "height":90
               },
               "medium":{
                  "url":"https://i.ytimg.com/vi/frn3WT15Dy8/mqdefault.jpg",
                  "width":320,
                  "height":180
               },
               "high":{
                  "url":"https://i.ytimg.com/vi/frn3WT15Dy8/hqdefault.jpg",
                  "width":480,
                  "height":360
               },
               "standard":{
                  "url":"https://i.ytimg.com/vi/frn3WT15Dy8/sddefault.jpg",
                  "width":640,
                  "height":480
               },
               "maxres":{
                  "url":"https://i.ytimg.com/vi/frn3WT15Dy8/maxresdefault.jpg",
                  "width":1280,
                  "height":720
               }
            },
            "channelTitle":"The Hill",
            "categoryId":"25",
            "liveBroadcastContent":"none",
            "defaultLanguage":"en-US",
            "localized":{
               "title":"'This Is Now Becoming A Pattern': Jordan Slams YouTube Covid-19 'Censorship'",
               "description":"During a Wednesday hearing on COVID-19 misinformation, Rep. Jim Jordan (R-OH) criticized YouTube for removing a video containing 'misinformation.' The Ohio Republican said the social media company should not be censoring doctors over 'science.'"
            },
            "defaultAudioLanguage":"en-US"
         },
         "contentDetails":{
            "duration":"PT6M19S",
            "dimension":"2d",
            "definition":"hd",
            "caption":"false",
            "licensedContent":true,
            "contentRating":{

            },
            "projection":"rectangular"
         },
         "statistics":{
            "viewCount":"989839",
            "likeCount":"35804",
            "dislikeCount":"798",
            "favoriteCount":"0",
            "commentCount":"7034"
         }
      }
   ],
   "pageInfo":{
      "totalResults":1,
      "resultsPerPage":1
   }
}

In order to get the title and subscriber count of the channel owner, we need to make another request to the channels API. We append the channel_ID to the querystring2 and then make a request to the channels API. We then proceed to render the context as the final response on our HTML template that we shall create below.

Youtube Template

On the exam/templates/ folder, create a youtube.html file and add the following lines of code:

{% load humanize %}
html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link
      rel="stylesheet"
      href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.0/css/bootstrap.min.css"
    />
    <style>
    @import url(https://fonts.googleapis.com/css2?family=Roboto&display=swap);*{margin:0;padding:0;box-sizing:border-box}body{background-color:rgba(200,200,200,.1);font-family:Roboto,sans-serif}.icon-spacing{padding:0 10px}.flex-column{flex-direction:column}.space-between{justify-content:space-between}.grid-container{position:relative;left:250px}nav{align-items:center;position:fixed;top:0;width:100%;height:50px;background-color:#fff;justify-content:space-between;z-index:1}#left-navbar{justify-content:space-evenly;width:12%;align-items:center;margin-left:15px}#mid-navbar{align-self:center;margin-right:20px;width:50%;justify-content:center}#search-bar{background-color:#fff;width:85%;padding:3px 10px;border:1px solid silver;border-radius:3px 0 0 3px;font-size:medium}#search-button{border:1px solid gray;padding:0 30px;border-radius:0 3px 3px 0}#right-navbar{align-items:center;width:120px;justify-content:space-between;margin-right:50px}.main-video{margin-left:initial}.flex{display:flex}.upcoming{font-family:Roboto,sans-serif;font-size:16px;color:#030303}.input{width:550px;position:relative;left:219px}.container{position:relative;top:85px;left:22px}.main-fc{width:60%;margin-left:30px}.other-videos{width:30%}#autobutton{border:none;background-color:transparent}.sidebar{position:relative;top:80px}.image{float:left;height:10px;position:relative;width:168px}.aside-video{grid-template-columns:180px 180px;display:grid;margin-bottom:4px}.song-title{font-size:14px;color:#030303;display:inline-block}.aside-description{position:relative;bottom:8px;font-size:12px}.artist-name{font-size:13px;color:#606060}.aside-views{font-size:13px;color:#606060}.video-duration{position:relative;float:right;bottom:18px;right:14px;font-size:12px;background-color:#000;color:#f5f5f5}.video-title{font-size:1.15rem;font-weight:300}.images{border-radius:50%;width:40px;height:40px}.smaller-gray{font-size:.9rem;color:gray}#video-below{justify-content:space-between;align-items:center}.video-section{justify-self:start}.subscribe-button{float:right;margin-top:-60px;background-color:#c00;font-size:smaller;border:none;color:#fff;padding:10px}.align-center{align-items:center;align-content:center;align-self:center}.inline{display:inline}.upnext{justify-content:center}.sidebar-attributes{font-family:Roboto,sans-serif;line-height:1.2em}.page-profile p{font-size:14px}#Squatty-potty{margin-bottom:-5px;padding-bottom:5px;margin-top:-20px}#avatar{margin-right:16px;margin-top:-40px;width:50px;height:50px}.user-name-bolded{margin:0 10px;font-size:small}.comment-description{margin-left:50px;margin-top:-20px;font-size:smaller}.smallest-gray{font-size:.7rem;color:gray}.reply{margin-left:10px;font-weight:300}.replies{margin-top:10px}.gray{color:#636363}.sort-button{display:grid;grid-template-columns:1fr 4.5fr}.sort-by{margin-left:10px}button{border:none;background-color:transparent}
    style>
    <link
      rel="stylesheet"
      href="https://stackpath.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css"
    />
    <title>Youtube Pagetitle>
  head>

  <body>
    <nav class="flex">
      <div id="left-navbar" class="flex">
        <a href="#"><img src="https://res.cloudinary.com/prometheusapi/image/upload/v1638303419/hamburger_v2ird3.png" alt="" />a>
        <a href="#"><img src="https://res.cloudinary.com/prometheusapi/image/upload/v1638303419/youtubelogos_kkvpsz.png" alt="" />a>
      div>
      <div id="mid-navbar" class="flex">
        <input id="search-bar" type="text" placeholder="Search" />
        <button id="search-button">
          <i class="fa fa-search" aria-hidden="true">i>
        button>
      div>

      <div id="right-navbar" class="flex">
        <a href="#"><img src="https://res.cloudinary.com/prometheusapi/image/upload/v1638303419/nav1_ngreqx.png" alt="" />a>
        <a href="#"><img src="https://res.cloudinary.com/prometheusapi/image/upload/v1638303419/nav2_zhlzur.png" alt="" />a>
        <a href="#"><img src="https://res.cloudinary.com/prometheusapi/image/upload/v1638303419/nav4_oc1mlz.png" alt="" />a>
        <a href="#"><img src="https://res.cloudinary.com/prometheusapi/image/upload/v1638303419/nav3_ayeunv.png" alt="" />a>
      div>
    nav>
    {% for data in items %}
    <main class="d-flex space-between">
      <div class="main-fc d-flex flex-column">
        <div class="container">
          <div class="main-video">
            <iframe
              width="853"
              height="480"
              src="https://www.youtube.com/embed/{{videoid}}"
              allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
              allowfullscreen
            >iframe>
            <br>
            <br>
            <h3 class="video-title">
              {{data.snippet.title}}
            h3>
            <div id="video-below" class="flex flex-horizontal">
              <div class="video-views">
                <p class="smaller-gray">{{data.statistics.viewCount|intcomma}} views • {{data.snippet.publishedAt|slice:"10"}}p>
              div>
              <div class="flex">
                <button>
                  <i class="fa fa-thumbs-up icon-spacing" aria-hidden="true">
                    {{data.statistics.likeCount|intcomma}}i
                  >
                button>

                <button>
                  <i class="fa fa-thumbs-down icon-spacing" aria-hidden="true">
                    {{data.statistics.dislikeCount|intcomma}}i
                  >
                button>
                <button><i class="fa fa-share icon-spacing">SHAREi>button>
                <button>
                  <i class="fa fa-save icon-spacing" aria-hidden="true">
                    SAVEi
                  >
                button>
                <button>
                  <i
                    class="fa fa-ellipsis-h icon-spacing"
                    aria-hidden="true"
                  >i>
                button>
              div>
            div>
            <hr />
            <br />
          div>

          <div class="page-profile">
            <div class="user-description">
              <button class="subscribe-button">ALIENXbutton>
            div>
            <h3>
              {{data.snippet.channelTitle}}
            h3>
            <p>
              {{data.snippet.description}}
            p>
          div>

          <div class="video-comments">
            <div class="comment-section">
              <hr />
              <div class="sort-button">
                <h6>{{data.statistics.commentCount|intcomma}} Commentsh6>
                <div>
                  {% endfor %}
                div>
              div>
              <i class="fa fa-sort-amount-down gray" aria-hidden="true">i>
            div>
            <hr />

          div>
        div>
      div>

    main>
  body>
html>

Template Explanation.

In the code above, we start by importing a very important package called Django Humanize. This package is useful for adding a “human touch” to data, in our case, changing the data pulled from youtube. The data we want to "humanize" in this case is the counts. You will notice from the API request above, the counts do not have commas, we then add the code below to format the data into commas.

<i class="fa fa-thumbs-up icon-spacing" aria-hidden="true">
                    {{data.statistics.likeCount|intcomma}}
</i>

Testing our API

Make necessary migrations and then run the server. Head over to a youtube video of your choice and remove the HTTPS to HTTP. Remove www.youtube.com to point to your localhost and you should end up with a URL that resembles:

If you have any questions, feel free to ask or test my API. Hit a thumbs down, sorry, thumbs up, if you enjoyed this post.

Build an Image Classification API Using Django Rest Framework.

Paul Kiragu — Tue, 23 Nov 2021 23:45:01 GMT

Overview:

Machine Learning (ML) and data science applications are in high demand. When ML algorithms offer information before it is known, the benefits for business are significant. Integrating machine learning algorithms for inference into production systems is a technological barrier hence the need for deploying ML models as API'S.

Introduction

In this short article, we implement AI based models to detect COVID-19 in Chest X-rays and CT Scans using four Deep Learning Algorithms: VGG16, ResNet50, InceptionV3 and Xception. Note, we shall focus mainly on implementing the api and not the model creation.

To test my live endpoint, send a POST request to the following URL with an x-ray image appended to the body of the request. You will get the following sample output if request was successful:

https://alienx.tech/api/v1/xray # for the x-ray images
https://alienx.tech/api/v1/ct # for CT scans

{
    "status": "success",
    "data": {
        "asset_id": "3e978ba830fb266978af20f2bf816f5e",
        "public_id": "vacpxfywfohgfprwhrso",
        "version": 1637699139,
        "version_id": "c9017c7d3b28ce797edefec0b0d72796",
        "signature": "e9c632b832e773cbbcb8906f93aba1d9e859d4bf",
        "width": 1205,
        "height": 1395,
        "format": "png",
        "resource_type": "image",
        "created_at": "2021-11-23T20:25:39Z",
        "tags": [],
        "bytes": 1325222,
        "type": "upload",
        "etag": "86005d3c34202b10949db5569570cd16",
        "placeholder": false,
        "url": "http://res.cloudinary.com/prometheusapi/image/upload/v1637699139/vacpxfywfohgfprwhrso.png",
        "secure_url": "https://res.cloudinary.com/prometheusapi/image/upload/v1637699139/vacpxfywfohgfprwhrso.png",
        "original_filename": "covid-19-pneumonia-22",
        "api_key": "138196782467569"
    },
    "url": "http://res.cloudinary.com/prometheusapi/image/upload/v1637699139/vacpxfywfohgfprwhrso.png",
    "xception_chest_pred": "100.00% COVID",
    "inception_chest_pred": "100.00% COVID",
    "vgg_chest_pred": "100.00% COVID",
    "resnet_chest_pred": "100.00% COVID"
}

ML Model Building

The dataset for the project was gathered from two open source Github repositories:

Chest X-ray images (1000 images) were obtained from: https://github.com/ieee8023/covid-chestxray-dataset
CT Scan images (750 images) were obtained from: https://github.com/UCSD-AI4H/COVID-CT/tree/master/Data-split

Four algorithms: VGG16, ResNet50, InceptionV3 and Xception were trained separately on Chest X-rays and CT Scans, giving us a total of 8 deep learning models. 80% of the images were used for training the models and the remaining 20% for testing the accuracy of the models.

The code for training the 8 models is available on the creators github or my repository. The model for the project can be found on the following google drive.

Turning the Model into an RESTFUL API

Following Python best practices, we will create a virtual environment for our project, and install the required packages.

First, create the project directory.

$ mkdir djangoapp
$ cd djangoapp

Now, create a virtual environment and install the required packages.

For macOS and Unix systems:

$ python3 -m venv myenv
$ source myenv/bin/activate
(myenv) $ pip install django requests djangorestframework tensorflow cloudinary opencv-python

For Windows:

$ python3 -m venv myenv
$ myenv\Scripts\activate
(myenv) $ pip install django requests djangorestframework tensorflow cloudinary opencv-python

Setting Up Your Django Application

First, navigate to the directory djangoapp we created and establish a Django project.

(myenv) $ django-admin startproject mainapp

This will auto-generate some files for your project skeleton:

mainapp/
    manage.py
    mainapp/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Now, navigate to the directory you just created (make sure you are in the same directory as manage.py) and create your app directory.

(myenv) $ python manage.py startapp monitor

This will create the following:

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
        __init__.py
    models.py
    tests.py
    views.py

On the mainapp/settings.py file, look for the following line and add the app we just created above.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',#new line
    'monitor', #new line
]

Ensure you are in the monitor directory then create a new directory called templates and a new file called urls.py. Your directory structure of monitor application should look like this

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
    templates/
        __init__.py
    models.py
    tests.py
    urls.py
    views.py

Ensure your mainapp/urls.py file, add our monitor app URL to include the URLs we shall create next on the monitor app:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    #path('admin/', admin.site.urls),
    path('', include('monitor.urls')),#monitor app url
]

Now, on the monitor/urls.py file, add our website there:

from django.urls import path
from .views import *

urlpatterns = [
    path('api/upload/xray', UploadView.as_view(), name = 'prediction'),
    path('api/upload/ct', CTUploadView.as_view(), name = 'ct_prediction'),
]

(venv)$ mkdir ml
(venv)$ mkdir ml/models
(venv)$ mkdir ml/data

We also need to tell Django where our machine learning model is located and also add our cloudinary configuration there. Add these lines to settings.py file:

import os
import cloudinary

cloudinary.config( 
  cloud_name = "prometheusapi", 
  api_key = "GETYOURAPIKEY", 
  api_secret = "GETYOURAPIKEY" 
)

MODELS = os.path.join(BASE_DIR, 'ml/models')

Load Keras Model through apps.py

Load your machine learning models in apps.py so that when the application starts, the trained model is loaded only once. Otherwise, the trained model is loaded each time an endpoint is called, and then the response time will be slower.

Let’s update apps.py

import os
from django.apps import AppConfig
from django.conf import settings
from tensorflow.keras.models import load_model
from tensorflow import keras


class ResNetModelConfig(AppConfig):
    name = 'resnetAPI'
    MODEL_FILE = os.path.join(settings.MODELS, "resnet_chest.h5")
    model = keras.models.load_model(MODEL_FILE)

class ResNetCTModelConfig(AppConfig):
    name = 'resnetCTAPI'
    MODEL_FILE = os.path.join(settings.MODELS, "resnet_ct.h5")
    model = keras.models.load_model(MODEL_FILE)

class VGGModelConfig(AppConfig):
    name = 'vggAPI'
    MODEL_FILE = os.path.join(settings.MODELS, "vgg_chest.h5")
    model = keras.models.load_model(MODEL_FILE)

class VGGCTModelConfig(AppConfig):
    name = 'vggCTAPI'
    MODEL_FILE = os.path.join(settings.MODELS, "vgg_ct.h5")
    model = keras.models.load_model(MODEL_FILE)    

class InceptionModelConfig(AppConfig):
    name = 'inceptionv3_chestAPI'
    MODEL_FILE = os.path.join(settings.MODELS, "inceptionv3_chest.h5")    
    model = keras.models.load_model(MODEL_FILE)

class InceptionCTModelConfig(AppConfig):
    name = 'inceptionv3_chestCTAPI'
    MODEL_FILE = os.path.join(settings.MODELS, "inception_ct.h5")    
    model = keras.models.load_model(MODEL_FILE)    

class ExceptionModelConfig(AppConfig):
    name = 'xception_chestAPI'
    MODEL_FILE = os.path.join(settings.MODELS, "xception_chest.h5")    
    model = keras.models.load_model(MODEL_FILE)

class ExceptionCTModelConfig(AppConfig):
    name = 'xception_chestCTAPI'
    MODEL_FILE = os.path.join(settings.MODELS, "xception_ct.h5")    
    model = keras.models.load_model(MODEL_FILE)

Edit views.py

The last step is to update views.py. The views will be mainly responsible for two tasks:

Process incoming POST requests.
Make a prediction with the incoming data and give the result as a Response.

import urllib
from django.shortcuts import render
import numpy as np
from .apps import *
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser, JSONParser
import cloudinary.uploader
import matplotlib.pyplot as plt
import cv2

# Create your views here.
class UploadView(APIView):
    parser_classes = (
        MultiPartParser,
        JSONParser,
    )

    @staticmethod
    def post(request):
        file = request.data.get('picture')
        upload_data = cloudinary.uploader.upload(file)
        #print(upload_data)
        img = upload_data['url']


        #load models
        resnet_chest = ResNetModelConfig.model
        vgg_chest = VGGModelConfig.model
        inception_chest = InceptionModelConfig.model
        xception_chest = ExceptionModelConfig.model

        req = urllib.request.urlopen(img)
        arr = np.asarray(bytearray(req.read()), dtype=np.uint8)
        image = cv2.imdecode(arr, -1) # 'Load it as it is'
        #image = cv2.imread('upload_chest.jpg') # read file 
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # arrange format as per keras
        image = cv2.resize(image,(224,224))
        image = np.array(image) / 255
        image = np.expand_dims(image, axis=0)

        resnet_pred = resnet_chest.predict(image)
        probability = resnet_pred[0]
        #print("Resnet Predictions:")
        if probability[0] > 0.5:
            resnet_chest_pred = str('%.2f' % (probability[0]*100) + '% COVID') 
        else:
            resnet_chest_pred = str('%.2f' % ((1-probability[0])*100) + '% NonCOVID')
        #print(resnet_chest_pred)

        vgg_pred = vgg_chest.predict(image)
        probability = vgg_pred[0]
        #print("VGG Predictions:")
        if probability[0] > 0.5:
            vgg_chest_pred = str('%.2f' % (probability[0]*100) + '% COVID') 
        else:
            vgg_chest_pred = str('%.2f' % ((1-probability[0])*100) + '% NonCOVID')
        #print(vgg_chest_pred)

        inception_pred = inception_chest.predict(image)
        probability = inception_pred[0]
        #print("Inception Predictions:")
        if probability[0] > 0.5:
            inception_chest_pred = str('%.2f' % (probability[0]*100) + '% COVID') 
        else:
            inception_chest_pred = str('%.2f' % ((1-probability[0])*100) + '% NonCOVID')
        #print(inception_chest_pred)

        xception_pred = xception_chest.predict(image)
        probability = xception_pred[0]
        #print("Xception Predictions:")
        if probability[0] > 0.5:
            xception_chest_pred = str('%.2f' % (probability[0]*100) + '% COVID') 
        else:
            xception_chest_pred = str('%.2f' % ((1-probability[0])*100) + '% NonCOVID')
        #print(xception_chest_pred)
        return Response({
            'status': 'success',
            'data': upload_data,
            'url':img,
            'xception_chest_pred':xception_chest_pred,
            'inception_chest_pred':inception_chest_pred,
            'vgg_chest_pred':vgg_chest_pred,
            'resnet_chest_pred':resnet_chest_pred,
        }, status=201)


class CTUploadView(APIView):
    parser_classes = (
        MultiPartParser,
        JSONParser,
    )

    @staticmethod
    def post(request):
        file = request.data.get('picture')
        upload_data = cloudinary.uploader.upload(file)
        #print(upload_data)
        img = upload_data['url']


        #load models
        resnet_chest = ResNetCTModelConfig.model
        vgg_chest = VGGCTModelConfig.model
        inception_chest = InceptionCTModelConfig.model
        xception_chest = ExceptionCTModelConfig.model

        req = urllib.request.urlopen(img)
        arr = np.asarray(bytearray(req.read()), dtype=np.uint8)
        image = cv2.imdecode(arr, -1) # 'Load it as it is'
        #image = cv2.imread('upload_chest.jpg') # read file 
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # arrange format as per keras
        image = cv2.resize(image,(224,224))
        image = np.array(image) / 255
        image = np.expand_dims(image, axis=0)

        resnet_pred = resnet_chest.predict(image)
        probability = resnet_pred[0]
        #print("Resnet Predictions:")
        if probability[0] > 0.5:
            resnet_chest_pred = str('%.2f' % (probability[0]*100) + '% COVID') 
        else:
            resnet_chest_pred = str('%.2f' % ((1-probability[0])*100) + '% NonCOVID')
        #print(resnet_chest_pred)

        vgg_pred = vgg_chest.predict(image)
        probability = vgg_pred[0]
        #print("VGG Predictions:")
        if probability[0] > 0.5:
            vgg_chest_pred = str('%.2f' % (probability[0]*100) + '% COVID') 
        else:
            vgg_chest_pred = str('%.2f' % ((1-probability[0])*100) + '% NonCOVID')
        #print(vgg_chest_pred)

        inception_pred = inception_chest.predict(image)
        probability = inception_pred[0]
        #print("Inception Predictions:")
        if probability[0] > 0.5:
            inception_chest_pred = str('%.2f' % (probability[0]*100) + '% COVID') 
        else:
            inception_chest_pred = str('%.2f' % ((1-probability[0])*100) + '% NonCOVID')
        #print(inception_chest_pred)

        xception_pred = xception_chest.predict(image)
        probability = xception_pred[0]
        #print("Xception Predictions:")
        if probability[0] > 0.5:
            xception_chest_pred = str('%.2f' % (probability[0]*100) + '% COVID') 
        else:
            xception_chest_pred = str('%.2f' % ((1-probability[0])*100) + '% NonCOVID')
        #print(xception_chest_pred)
        return Response({
            'status': 'success',
            'data': upload_data,
            'url':img,
            'xceptionCT_chest_pred':xception_chest_pred,
            'inceptionCT_chest_pred':inception_chest_pred,
            'vggCT_chest_pred':vgg_chest_pred,
            'resnetCT_chest_pred':resnet_chest_pred,
        }, status=201)

Testing our API

Create the necessary migrations then run the server:

(myenv) $ python manage.py makemigrations
(myenv) $ python manage.py migrate
(myenv) $ python manage.py runserver

Fire up Postman and make a POST request with an image appended to the body.

Thanks for staying tuned!

Develop a Spam Filtering Model in Python & Deploy it with Django

Paul Kiragu — Wed, 17 Nov 2021 15:30:02 GMT

Introduction

Spam is a fact of life on the internet. If you enable comments or contact sections on your website, you will have to deal with spammers. To prevent your site from making a poor first impression, you’ll need to find a way to stop spam in its tracks. This is especially important if you are developing a website without a content management system like Wordpress as they come bundled with spam filtering plugins. You could also use an API like Akismet, however this comes at a cost, which can be avoided by implementing a relatively accurate model of your own.

Kaggle and other data science bootcamps are great for learning how to build and optimize models, but they don’t teach you how to actually use this models in real world scenarios, where there’s a major difference between building a model, and deploying it to be used by end users on the internet.

In this tutorial, you’re going to build an SMS spam detection web application. This application will be built in Python using the Django framework, and will include a deep learning model that you will train to detect SMS spam by leveraging the Naive Bayes theorem.

Naive Bayes classification

The classification of Naive Bayes is a simple probability algorithm based on the fact that all model characteristics are independent. We assume that every word in the message is independent of all other words in the context of the spam filters, and we count them with the ignorance of the context.

By the state of the current set of terms, our classification algorithm generates probabilities of the message to be spam or not spam. The probability estimation is based on the Bayes formula, and the formula components are determined on the basis of the word frequencies in the whole message package.

Model Building

The data is a collection of SMS messages tagged as spam or ham that can be found here. First, we will use this dataset to build a prediction model that will accurately classify which texts are spam and which are not, then save the model to be used later for predictions.

Exploration of dataset

The first thing that should be done is to import dependencies. If you do not have the libraries installed, kindly do so before proceeding.

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import classification_report
import joblib
import pickle

Next, we load the dataset using pandas:

df = pd.read_csv('https://raw.githubusercontent.com/paulwababu/datasets/main/spam.csv', encoding = 'latin-1')
print(df.head())

Drop the unwanted columns, like so:

df.drop(['Unnamed: 2', 'Unnamed: 3', 'Unnamed: 4'], axis=1, inplace=True)

We have to convert the non-numerical column 'spam' and 'ham' into numerical values using pandas map() function

df['label'] = df['v1'].map({'ham': 0, 'spam': 1})

Then we have to separate the feature columns(independent variables) from the target column(dependent variable).

The feature columns are the columns that we try to predict from, and the target column is the column with the values we try to predict.

X = df['v2'] 
y = df['label']

ML Model Building

Let us now proceed to building our actual model.

cv = CountVectorizer()
X = cv.fit_transform(X) 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
model = MultinomialNB()
model.fit(X_train,y_train)
#model.score(X_test,y_test)
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))

              precision    recall  f1-score   support

           0       0.99      0.99      0.99      1587
           1       0.93      0.92      0.92       252

    accuracy                           0.98      1839
   macro avg       0.96      0.95      0.96      1839
weighted avg       0.98      0.98      0.98      1839

Not only Naive Bayes classifier easy to implement but also provides very good result.

In the code above, we create a vectorize function that transforms a given text into a vector on the basis of the frequency (count) of each word that occurs in the entire text. We then proceed to splitting the data into train and test variables which we use to get the classification report of the model. We then call the multinomial Naive Bayes model which is suitable for classification with discrete features (e.g., word counts for text classification)

Model and Vectorizer Persistence.

After training the model, we should to have a way to persist the model for future use without having to retrain. To achieve this, need to save the model for the later use. Add the following lines of code:

# Save the model
joblib_file = "MultinomialNaiveBayesModel.joblib"
joblib.dump(model, joblib_file)

We also need to save the vectorize function that we created earlier otherwise you throw it away because a vectorizer once created, doesn't exist past the lifetime of your vectorize function.

# Save the vectorizer
vec_file = 'MultinomialNaiveBayesModelVectorizer.pickle'
pickle.dump(cv, open(vec_file, 'wb'))

If we intend to retrain the model, we can use the partial_fit function in order to keep improving the model incase of model degradation over time. I will post a blog later that addresses how to identify and correct dataset shift in machine learning

Turning the Spam Message Classifier into a Django Web Application

Having trained and saved the model for classifying SMS messages in the previous section, we will develop a web application that consists of a simple web page with a form field that lets us enter a message. After submitting the message to the web application, it will render it on a new page which gives us a result of spam or not spam.

Below is snapshot of the final implementation

Following Python best practices, we will create a virtual environment for our project, and install the required packages.

First, create the project directory.

$ mkdir djangoapp
$ cd djangoapp

Now, create a virtual environment and install the required packages.

For macOS and Unix systems:

$ python3 -m venv myenv
$ source myenv/bin/activate
(myenv) $ pip install django requests numpy joblib scikit-learn

For Windows:

$ python3 -m venv myenv
$ myenv\Scripts\activate
(myenv) $ pip install django requests numpy joblib scikit-learn

Setting Up Your Django Application

First, navigate to the directory djangoapp we created and establish a Django project.

(myenv) $ django-admin startproject mainapp

This will auto-generate some files for your project skeleton:

mainapp/
    manage.py
    mainapp/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Now, navigate to the directory you just created (make sure you are in the same directory as manage.py) and create your app directory.

(myenv) $ python manage.py startapp monitor

This will create the following:

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
        __init__.py
    models.py
    tests.py
    views.py

On the mainapp/settings.py file, look for the following line and add the app we just created above.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'monitor', #new line
]

Ensure you are in the monitor directory then create a new directory called templates then a new file called urls.py. Your directory structure of monitor application should look like this

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
    templates/
        __init__.py
    models.py
    tests.py
    urls.py
    views.py

Ensure your mainapp/urls.py file, add our monitor app URL to include the URLs we shall create next on the monitor app:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    #path('admin/', admin.site.urls),
    path('', include('monitor.urls')),#monitor app url
]

Now, on the monitor/urls.py file, add our website there:

from django.urls import path
from .views import *

urlpatterns = [
    path('', views.sms, name = 'sms'),
    path('inbox/', views.inbox, name='inbox')
]

(venv)$ mkdir ml
(venv)$ mkdir ml/models
(venv)$ mkdir ml/data

We also need to tell Django where our machine learning model and our vectorizer file is located. Add these lines to settings.py file:

import os
MODELS = os.path.join(BASE_DIR, 'ml/models')

Load Model and Vectorizer through apps.py

Load your machine learning models and your vectorizer in apps.py so that when the application starts, the trained model is loaded only once. Otherwise, the trained model is loaded each time an endpoint is called, and then the response time will be slower.

Let’s update apps.py

import os
import joblib
from django.apps import AppConfig
from django.conf import settings


class ApiConfig(AppConfig):
    name = 'api'
    MODEL_FILE = os.path.join(settings.MODELS, "MultinomialNaiveBayesModel.joblib")
    model = joblib.load(MODEL_FILE)

class VectorizerConfig(AppConfig):
    name = 'api2'
    MODEL_FILE = os.path.join(settings.MODELS, "MultinomialNaiveBayesModelVectorizer.pickle")
    model = joblib.load(MODEL_FILE)

Edit models.py

Create our database models which we shall use to store our classified models. On the monitor/models.py file:

from django.db import models

# Create your models here.
class Monitor2(models.Model):
    message = models.CharField(max_length=50, blank=True, null=True)
    SPAM = 1
    HAM = 0
    IS_SPAM_OR_NAH = [(SPAM, 'spam'), (HAM, 'not_spam')]
    messageClassified = models.IntegerField(choices=IS_SPAM_OR_NAH, null=True)
    contact = models.CharField(max_length=50, blank=True, null=True)

Edit views.py

The views will be mainly responsible for two tasks:

Process incoming POST requests.
Make a prediction with the incoming data and give the result as a Response.
Display the classified text into a HTML template.

import os
from datetime import datetime
from .models import *
from django.shortcuts import render, redirect


def sms(request):
    if request.method == 'POST':
        number = request.POST['contact']
        message = request.POST['message']
# datetime object containing current date and time
        now = datetime.now()    
        now = now.strftime("%d/%m/%Y %H:%M:%S")

        naiveModel = ApiConfig.model
        naiveVect = VectorizerConfig.model
        convertString = str(message)
        message = convertString
        data = [message]
        vect = naiveVect.transform(data).toarray()
        my_prediction = naiveModel.predict(vect)
        print(my_prediction)
        saveNow = Monitor2(
            message=message,
            messageClassified=my_prediction,
            contact=number
        )
        saveNow.save()
    return render(request, 'sms.html')

#inbox view
def inbox(request):
    dataSaved = Monitor2.objects.all()
    data = {
        "dataSaved": dataSaved,
    }
    print(data)
    return render(request, 'inbox.html', data)

On the monitor/templates folder, create sms.html and inbox.html web page and add the lines below:

monitor/templates/sms.html file:

html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Documenttitle>
head>
<body>
    <form id="myform" method="POST">
        {% csrf_token %}
          <div class="row">
              <div class="col-6 form-group">
                  <input type="text" name="name1" class="form-control p-4" placeholder="Your Name" required="required">
              div>
              <div class="col-6 form-group">
                  <input type="text" name="contact" class="form-control p-4" placeholder="Your Contact" required="required">
              div>
          div>

          <div class="form-group">
              <textarea class="form-control py-3 px-4" name="message" rows="5" placeholder="Message" required="required">textarea>
          div>
          <div>
              <button class="btn btn-primary py-3 px-5" type="submit">Send Messagebutton>
          div>
      form>
body>
html>

monitor/templates/inbox.html file:

html>
<html>
<style>
table, th, td {
  border:1px solid black;
}
style>
<body>

<h2>A basic HTML tableh2>

<table style="width:100%">
    <tr>
        <th>#th>
        <th>Fromth>
        <th>Bodyth>
        <th>Classificationth>
    tr>
  {% for x in dataSaved %}
  <tr>
      <td>{{ loop.index }}td>
      <td>{{ x.contact }}td>
      <td>{{ x.message }}td>
      {% if x.messageClassified == 1 %}
      <td>Spamtd>
      {% else %}
      <td>Non Spamtd>
      {% endif %}
  tr>
  {% endfor %}
table>

body>
html>

Make the necessary migrations like so:

(myenv) $ python manage.py makemigrations
(myenv) $ python manage.py migrate
(myenv) $ python manage.py runserver

Testing if it works!

Head over http://127.0.0.1:8000 and complete the form with both spam and non spam

Proceed to http://127.0.0.1:8000/inbox to check out the classified data! Below is a snapshot of my implementation, sorry I couldn't make the CSS;(

Thanks for staying tuned!

Build a Machine Learning API With Django Rest Framework

Paul Kiragu — Wed, 10 Nov 2021 10:41:38 GMT

Introduction

Finding the proper treatment, especially drug treatment, can be a difficult task. Whether you're looking for drugs to treat seasonal allergies, or you're treating a serious illness, collecting data on various aspects of patients and what treatments they best respond to can be vital for saving lives and increasing patient convenience. Here, we'll act as a data analyst compiling medical data on a set of patients all suffering from the same illness. During their treatment, each patient responded to one of five drug treatments

In this article, we'll:

Build a simple Decision Tree model to predict Drug Treatment with relatively high accuracy.
Create an API with Django Rest Framework that will be used to consume the Decision Tree Model created.

Build a Model

Let us start by building and saving a machine learning model that will be later used to make predictions for our API.

Dataset

Find the dataset for this model on my Github repo. The file is a CSV with data from different patients who received treatment for the same disease. The different drugs can be drugA, drugB, drugC, drugX, or drugY.

Let’s get started with the exploration of the dataset.

The first thing that should be done is to import dependencies. If you do not have the libraries installed, kindly do so before proceeding.

import pandas
from sklearn import tree
import pydotplus
from sklearn.tree import DecisionTreeClassifier
import matplotlib.pyplot as plt
import matplotlib.image as pltimg
from sklearn.metrics import classification_report, plot_confusion_matrix, plot_roc_curve
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import joblib

Next, we load the dataset using pandas:

df = pandas.read_csv("https://raw.githubusercontent.com/paulwababu/datasets/main/drug200.csv")
print(df.head())

   Age Sex      BP Cholesterol  Na_to_K   Drug
0   23   F    HIGH        HIGH   25.355  drugY
1   47   M     LOW        HIGH   13.093  drugC
2   47   M     LOW        HIGH   10.114  drugC
3   28   F  NORMAL        HIGH    7.798  drugX
4   61   F     LOW        HIGH   18.043  drugY

To make a decision tree, all data has to be numerical.

We have to convert the non-numerical columns 'Gender', 'Cholesterol', and 'BP' into numerical values.

Pandas have a map() method that takes a dictionary with information on how to convert the values.

#Change string values into numerical values:
d = {'M': 1, 'F': 0}
df['Sex'] = df['Sex'].map(d)

d = {'HIGH': 0, 'LOW': 1, 'NORMAL': 2}
df['BP'] = df['BP'].map(d)

d = {'HIGH': 0, 'LOW': 1, 'NORMAL': 2}
df['Cholesterol'] = df['Cholesterol'].map(d)
print(df.head())

   Age  Sex  BP  Cholesterol  Na_to_K   Drug
0   23    0   0            0   25.355  drugY
1   47    1   1            0   13.093  drugC
2   47    1   1            0   10.114  drugC
3   28    0   2            0    7.798  drugX
4   61    0   1            0   18.043  drugY

Then we have to separate the feature columns(independent variables) from the target column(dependent variable).

The feature columns are the columns that we try to predict from, and the target column is the column with the values we try to predict.

#independent variables
features = ['Age', 'Sex', 'BP', 'Cholesterol', 'Na_to_K']
X = df[features]

#dependent variable
y = df['Drug']

     Age  Sex  BP  Cholesterol  Na_to_K
0     23    0   0            0   25.355
1     47    1   1            0   13.093
2     47    1   1            0   10.114
3     28    0   2            0    7.798
4     61    0   1            0   18.043
..   ...  ...  ..          ...      ...
195   56    0   1            0   11.567
196   16    1   1            0   12.006
197   52    1   2            0    9.894
198   23    1   2            2   14.020
199   40    0   1            2   11.349

[200 rows x 5 columns]
0      drugY
1      drugC
2      drugC
3      drugX
4      drugY
       ...  
195    drugC
196    drugC
197    drugX
198    drugX
199    drugX
Name: Drug, Length: 200, dtype: object

Now we can create the actual decision tree, fit it with our details, and save a .png file on the computer:

dtree = DecisionTreeClassifier()
dtree = dtree.fit(X, y)
data = tree.export_graphviz(dtree, out_file=None, feature_names=features)
graph = pydotplus.graph_from_dot_data(data)
graph.write_png('mydecisiontree2.png')

img=pltimg.imread('mydecisiontree2.png')
imgplot = plt.imshow(img)
plt.show()

Results Explained

The decision tree uses your earlier decisions. Let us read the different aspects of the decision tree:

Salt to Potassium ratio (Na_to_K)

Na_to_K <= 14.829 means that every patient with a sodium-potassium of 14.829 or lower will follow the True arrow (to the left), and the rest will follow the False arrow (to the right).

gini = 0.694 refers to the quality of the split, where 0.0 would mean all of the samples got the same result, and 0.5 would mean that the split is done exactly in the middle.

samples = 200 means that there are 200 patients left at this point in the decision, which is all of them since this is the first step.

value = [23, 16, 16, 54, 91] means that of these 200 patients, 23 will get "drugY", 16 will get "drugB", 16 will get "drugC", 54 will get "drugX" and 91 will get "drugY",

Note, to know these values, we ran:

print(df['Drug'].value_counts())

drugY    91
drugX    54
drugA    23
drugB    16
drugC    16
Name: Drug, dtype: int64

Predict Values

We can use the Decision Tree to predict new values.

Example: What prescription(among the five drugs: Y X A B and C) should a 68-year-old female with normal levels of cholesterol and BP and has salt to potassium level of 27.10 take?

print(dtree.predict([[68, 0, 2, 2, 27.10]]))
# ['drugY'] will be the output

The system predicts drugY. If you follow the decision tree implemented above, you will arrive at drugY too! Sorry, I cannot explain every aspect of the decision tree, it's quite lengthy and is not within the scope of this tutorial. Let us now proceed to save the model built

# Save the model
joblib_file = "DecisionTreeModel.joblib"
joblib.dump(dtree, joblib_file)

Now we can start the Django section.

Create an API With Django Rest Framework

Following Python best practices, we will create a virtual environment for our project, and install the required packages.

First, create the project directory.

$ mkdir djangoapp
$ cd djangoapp

Now, create a virtual environment and install the required packages.

For macOS and Unix systems:

$ python3 -m venv myenv
$ source myenv/bin/activate
(myenv) $ pip install django requests djangorestframework numpy joblib scikit-learn

For Windows:

$ python3 -m venv myenv
$ myenv\Scripts\activate
(myenv) $ pip install django requests djangorestframework numpy joblib scikit-learn

Setting Up Your Django Application

First, navigate to the directory djangoapp we created and establish a Django project.

(myenv) $ django-admin startproject mainapp

This will auto-generate some files for your project skeleton:

mainapp/
    manage.py
    mainapp/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Now, navigate to the directory you just created (make sure you are in the same directory as manage.py) and create your app directory.

(myenv) $ python manage.py startapp monitor

This will create the following:

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
        __init__.py
    models.py
    tests.py
    views.py

On the mainapp/settings.py file, look for the following line and add the app we just created above.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',#new line
    'monitor', #new line
]

Ensure you are in the monitor directory then create a new directory called templates and a new file called urls.py. Your directory structure of monitor application should look like this

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
    templates/
        __init__.py
    models.py
    tests.py
    urls.py
    views.py

Ensure your mainapp/urls.py file, add our monitor app URL to include the URLs we shall create next on the monitor app:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    #path('admin/', admin.site.urls),
    path('', include('monitor.urls')),#monitor app url
]

Now, on the monitor/urls.py file, add our website there:

from django.urls import path
from .views import *

urlpatterns = [
    path('', Prediction.as_view(), name = 'prediction'),
]

(venv)$ mkdir ml
(venv)$ mkdir ml/models
(venv)$ mkdir ml/data

We also need to tell Django where our machine learning model is located. Add these lines to settings.py file:

import os
MODELS = os.path.join(BASE_DIR, 'ml/models')

Load Model through apps.py

Let’s update apps.py

import os
import joblib
from django.apps import AppConfig
from django.conf import settings


class ApiConfig(AppConfig):
    name = 'api'
    MODEL_FILE = os.path.join(settings.MODELS, "DecisionTreeModel.joblib")
    model = joblib.load(MODEL_FILE)

Edit views.py

The last step is to update views.py. The views will be mainly responsible for two tasks:

Process incoming POST requests.
Make a prediction with the incoming data and give the result as a Response.

import numpy as np
import pandas as pd
from .apps import *
from rest_framework.views import APIView
from rest_framework.response import Response


class Prediction(APIView):
    def post(self, request):
        #data = request.data
        age= request.GET.get('age')
        gender = request.GET.get('gender')
        bp = request.GET.get('bp')
        cholesterol = request.GET.get('cholesterol')
        salt = request.GET.get('salt')
        dtree = ApiConfig.model
        #predict using independent variables
        PredictionMade = dtree.predict([[age, gender, cholesterol, bp, salt]])
        response_dict = {"Predicted drug": PredictionMade}
        print(response_dict)
        return Response(response_dict, status=200)

Test the API

To test our API, I’ll use requests library, which is the de facto standard for making HTTP requests in Python. You can also use Postman or Insomnia for making the requests.

Let’s predict what drug prescription(among the five drugs: Y X A B and C) should a 68-year-old female with normal levels of cholesterol and BP and has salt to potassium level of 27 take?

import requests
predict = [print(x) for x in requests.post("http://127.0.0.1:8000/?age=68&gender=0&bp=2&cholesterol=2&salt=27")]

{'Predicted drug': ['drugY']}

Thank you for staying tuned! In case of any questions, feel free to ask in the comment section below.

Introduction: What is Prometheus?

Paul Kiragu — Sun, 24 Oct 2021 23:19:40 GMT

One month ago, I was set out to build an AI system for my home like JARVIS AI from Mark Zuckerberg, inspired by Iron Man. I decided to name it Prometheus, because the myth still makes up the lower strata of my psyche;) The AI system also has a chat bot interface that is connected to the system for remote operations. The system also has some features that are in active development and therefore I cannot blog them for now eg, using a drone for perimeter scouting and controlling home devices like bulbs using Prometheus. Credits to Flying Emu for the Awesome Jarvis UI that I happily copy pasted and reverse engineered a bit to show responses of the user.

What can Prometheus Do?

Our virtual assistant has the following features-:

a.) Web User Interface

Support Voice Conversations
Voice controlled SMS reminders
Voice controlled Google Search
Voice controlled web penetration testing and network scanning
Voice controlled intrusion detection via an IP camera with SMS alerts of intruders image(can be specified to animals)
Voice controlled MPESA business to customer payments(b2c)
Voice controlled MPESA customer to business payments(c2b)
Voice controlled airtime purchase
Symptoms Tracker for medical diagnosis and recommendations.

b.) ChatBot User Interface

Support Text Conversations
Support automated web penetration testing.
Support domain resolving
Support MPESA business to business payments(b2b) by simply uploading an image of the till number and the system does a wired transfer automatically.(Depreciated)
Support MPESA customer to business transactions
Get latest news
Panic mode that sends current GPS location shared via whatsapp to primary contacts

I am currently in the process of buying smart home devices and a drone to integrate them to Prometheus. One major problem with smart home devices is that they lack common standards to communicate, making integration a challenge.

What is Prometheus built with?

Prometheus is basically a hub of API's that leverages a number of artificial intelligence techniques, including speech recognition and object detection. It is written in Python's Django REST Framework, Flask microframework and Javascript.

Sneak Peak Feature: Voice Enabled Penetration Testing

See you in the first episode on how to build Prometheus Speech to Text

Build a web traffic monitor with Django and IPStack

Paul Kiragu — Mon, 04 Oct 2021 19:54:28 GMT

In today's online world, it is important to know where your website traffic comes from in order to help website owners understand their visitors better. In this tutorial, we build a very basic traffic monitor, the traffic monitor will display details about visitors on the website. For every visitor, there will multiple columns such as time of visit, continent, country and city.

Getting started with IPStack API

IPStack offers a powerful, real-time IP to geolocation API capable of looking up accurate location data and assessing security threats originating from risky IP addresses. Results are delivered within milliseconds in JSON or XML format. Using the ipstack API you will be able to locate website visitors at first glance and adjust your user experience and application accordingly.

The documentation outlines in detail the API features, available options and integration guides in different programming languages.

Get a free API access key to proceed using this API.

Sample API Response

IPStack API responses come with comprehensive location-related data, currency-related data, timezone-related data, connection-related data and security-related data. Have a look at the example below.

Please note: For illustration purposes we have included both Hostname Lookup and the Security Module in the above API response. Please refer to the sections Hostname Lookup and Security Module for details.

{
  "ip": "134.201.250.155",
  "hostname": "134.201.250.155",
  "type": "ipv4",
  "continent_code": "NA",
  "continent_name": "North America",
  "country_code": "US",
  "country_name": "United States",
  "region_code": "CA",
  "region_name": "California",
  "city": "Los Angeles",
  "zip": "90013",
  "latitude": 34.0453,
  "longitude": -118.2413,
  "location": {
    "geoname_id": 5368361,
    "capital": "Washington D.C.",
    "languages": [
        {
          "code": "en",
          "name": "English",
          "native": "English"
        }
    ],
    "country_flag": "https://assets.ipstack.com/images/assets/flags_svg/us.svg",
    "country_flag_emoji": "🇺🇸",
    "country_flag_emoji_unicode": "U+1F1FA U+1F1F8",
    "calling_code": "1",
    "is_eu": false
  },
  "time_zone": {
    "id": "America/Los_Angeles",
    "current_time": "2018-03-29T07:35:08-07:00",
    "gmt_offset": -25200,
    "code": "PDT",
    "is_daylight_saving": true
  },
  "currency": {
    "code": "USD",
    "name": "US Dollar",
    "plural": "US dollars",
    "symbol": "$",
    "symbol_native": "$"
  },
  "connection": {
    "asn": 25876,
    "isp": "Los Angeles Department of Water & Power"
  },
  "security": {
    "is_proxy": false,
    "proxy_type": null,
    "is_crawler": false,
    "crawler_name": null,
    "crawler_type": null,
    "is_tor": false,
    "threat_level": "low",
    "threat_types": null
  }
}

Prerequisites

Following Python best practices, we will create a virtual environment for our project, and install the required packages.

First, create the project directory.

$ mkdir djangoapp
$ cd djangoapp

Now, create a virtual environment and install the required packages.

For macOS and Unix systems:

$ python3 -m venv myenv
$ source myenv/bin/activate
(myenv) $ pip install django requests

For Windows:

$ python3 -m venv myenv
$ myenv\Scripts\activate
(myenv) $ pip install django requests

Setting Up Your Django Application

First, navigate to the directory djangoapp we created and establish a Django project.

(myenv) $ django-admin startproject mainapp

This will auto-generate some files for your project skeleton:

mainapp/
    manage.py
    mainapp/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Now, navigate to the directory you just created (make sure you are in the same directory as manage.py) and create your app directory.

(myenv) $ python manage.py startapp monitor

This will create the following:

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
        __init__.py
    models.py
    tests.py
    views.py

On the mainapp/settings.py file, look for the following line and add the app we just created above.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'monitor', #new line
]

Ensure you are in the monitor directory then create a new directory called templates and a new file called urls.py. Your directory structure of monitor application should look like this

monitor/
    __init__.py
    admin.py
    apps.py
    migrations/
    templates/
        __init__.py
    models.py
    tests.py
    urls.py
    views.py

Ensure your mainapp/urls.py file, add our monitor app URL to include the URLs we shall create next on the monitor app:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    #path('admin/', admin.site.urls),
    path('', include('monitor.urls')),#monitor app url
]

Now, on the monitor/urls.py file, add our website there:

from django.urls import path, include
from . import views

urlpatterns = [
    path('', views.home, name="home"),
    path('monitor/', views.monitor, name="monitor"), #system monitor view to be created next
]

Create our database models which we shall use to store our website visitors information. On the monitor/models.py file:

from django.db import models

# Create your models here.
class Monitor(models.Model):
    continent = models.CharField(max_length=50, blank=True, null=True)
    country = models.CharField(max_length=50, blank=True, null=True)
    city = models.CharField(max_length=50, blank=True, null=True)
    capital = models.CharField(max_length=50, blank=True, null=True)
    datetime = models.DateField(max_length=50, blank=True, null=True)
    ip = models.CharField(max_length=50, blank=True, null=True)

    def __str__(self):
        return self.ip

Now, on the monitor/views.py file, add the lines below:

from django.shortcuts import render, redirect
import requests
import time
import psutil
import os
from datetime import datetime
from .models import Monitor
from urllib.parse import urlparse
from django.core.paginator import Paginator

#traffic monitor
def traffic_monitor(request):
    dataSaved = Monitor.objects.all().order_by('-datetime')
    # Getting loadover15 minutes 
    load1, load5, load15 = psutil.getloadavg()
    cpu_usage = int((load15/os.cpu_count()) * 100)
    ram_usage = int(psutil.virtual_memory()[2])
    p = Paginator(dataSaved, 100)
    #shows number of items in page
    totalSiteVisits = (p.count)
    #find unique page viewers & Duration
    pageNum = request.GET.get('page', 1)
    page1 = p.page(pageNum)
    #unique page viewers
    a = Monitor.objects.order_by().values('ip').distinct()
    pp = Paginator(a, 10)
    #shows number of items in page
    unique = (pp.count)
    #update time
    now = datetime.now()
    data = {
        "now":now,
        "unique":unique,
        "totalSiteVisits":totalSiteVisits,
        "cpu_usage": cpu_usage,
        "ram_usage": ram_usage,
        "dataSaved": page1,
    }
    return render(request, 'traffic_monitor.html', data)
#home page
def home(request):
    x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
    if x_forwarded_for:
        ip = x_forwarded_for.split(',')[0]
    else:
        ip = request.META.get('REMOTE_ADDR')
    response = requests.get('http://api.ipstack.com/'+ip+'?access_key=GETYOURACCESSKEY') #change from HTTP to HTTPS on the IPSTACK API if you have a premium account
    rawData = response.json()
    print(rawData) # print this out to look at the response
    continent = rawData['continent_name']
    country = rawData['country_name']
    capital = rawData['city']
    city = rawData['location']['capital']
    now = datetime.now()
    datetimenow = now.strftime("%Y-%m-%d %H:%M:%S")
    saveNow = Monitor(
        continent=continent,
        country=country,
        capital=capital,
        city=city,
        datetime=datetimenow,
        ip=ip
    )
    saveNow.save()
    return render(request, 'home.html')

In the code above on the home view, the code starts by getting the users IP address. On getting the visitors IP, we then make a request to IPStack with the IP address above appended to the request and store the results to the database. If you wish to collect more information, just add them on the models and add the variable like so above with the respective data you are collecting eg country.

For the traffic monitor view, the code starts buy getting all data on the Monitor table and renders the data on the HTML page which we shall create below. The code also collects the cpu and ram usage of the system. Let us now proceed to rendering the data above on our webpage.

On the monitor/templates folder, create traffic_monitor.html and home.html web page and add the lines below:

monitor/templates/traffic_monitor.html file:

html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <meta name="description" content="">
    <meta name="author" content="">
    <title>ALienx – Connectionstitle>
    <link rel="canonical" href="https://getbootstrap.com/docs/4.0/examples/album/">
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css">
    
    <link href="https://getbootstrap.com/docs/4.0/dist/css/bootstrap.min.css" rel="stylesheet">

    
    <link href="https://getbootstrap.com/docs/4.0/examples/album/album.css" rel="stylesheet">
  head>

  <body>

    <header>
      <div class="collapse bg-dark" id="navbarHeader">
        <div class="container">
          <div class="row">
            <div class="col-sm-8 col-md-7 py-4">
              <h4 class="text-white">Abouth4>
              <p class="text-muted">Add some information about the album below, the author, or any other background context. Make it a few sentences long so folks can pick up some informative tidbits. Then, link them off to some social networking sites or contact information.p>
            div>
            <div class="col-sm-4 offset-md-1 py-4">
              <h4 class="text-white">Contacth4>
              <ul class="list-unstyled">
                <li><a href="#" class="text-white">Follow on Twittera>li>
                <li><a href="#" class="text-white">Like on Facebooka>li>
                <li><a href="#" class="text-white">Email mea>li>
              ul>
            div>
          div>
        div>
      div>

      <div class="navbar">
        <div class="container d-flex justify-content-between">
          <a href="#" class="navbar-brand d-flex align-items-center">
            <i class="fa fa-desktop" aria-hidden="true">i> <strong> Site Visitorsstrong>
          a>
          <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarHeader" aria-controls="navbarHeader" aria-expanded="false" aria-label="Toggle navigation">
            <span class="navbar-toggler-icon">span>
          button>
        div>
      div>
    header>

    <main role="main">

      <section class="jumbotron text-center">
        <div class="container">
          <h1 class="jumbotron-heading"><i class="fa fa-desktop" aria-hidden="true">i> Site Visitors and Resources Monitorh1>
          <p class="lead text-muted">Contact <a href="#">developera> if CPU & RAM USAGE exceeds 80%p>

        div>
      section>


      <div class="album py-7 bg-light">
            <div class="content-wrapper">
                <div class="container-fluid">

                  <ol class="breadcrumb">
                    <li class="breadcrumb-item">
                      <a href="#">Dashboarda>
                    li>
                    <li class="breadcrumb-item active">Site Traffic & Resources Monitorli>
                  ol>
                  <br>

                  
                  <div class="container">
                    <div class="row">
                        <div class="col-sm-3 block">
                            <div class="circle a">


                                <p><b>CPU Usageb>
                                    <br>
                                    <b>{{ cpu_usage }}%b>
                                p>

                            div>
                        div>
                        <div class="col-sm-3 block">
                            <div class="circle b">

                                <p><b>RAM Usageb> 
                                    <br>
                                    <b>{{ ram_usage }}%b>
                                p>

                            div>
                        div>

                        <div class="col-sm-3 block">
                            <div class="circle c">


                                <p><b>Total Site Visitsb> 
                                    <br>
                                    {{ totalSiteVisits }}
                                p>

                            div>
                        div>
                        <div class="col-sm-3 block">
                            <div class="circle d">
                                <p><b>Unique Page Viewsb>
                                    <br>
                                    {{ unique }}
                                p>
                            div>
                        div>


                    div>
                div>
                <style>
                  .block {
                                    bottom: 30px;
    text-align: center;
    vertical-align: middle;
}
.circle {
    font-size: 15px;
    background: white;
    border-radius: 200px;
    color: white;
    height: 170px;
    font-weight: bolder;
    width: 170px;
    display: table;
    margin: 20px auto;
}
.circle p {
    vertical-align: middle;
    display: table-cell;
}
                    .a {
                        background-color: #39c0dc;
                    }
                    .b {
                        background-color: darkorange;
                    }
                    .c {
                        background-color: #5c6dfd;
                    }
                    .d {
                        background-color: #f14c6e;
                    }
                    .g {
                        background-color: greenyellow;
                    }
                    .f{
                        background-color: fuchsia;
                    }
                style>

                  
                  <div class="card mb-3">
                    <div class="card-header">
                      <b style="color: #0664c9;"><i class="fa fa-table">i> User Sessionsb>div>
                    <div class="card-body">
                      <div class="table-responsive">
                        <table class="table table-bordered" id="contextTable" width="100%" cellspacing="0">
                          <thead>
                            <tr>
                              <th>Timeth>
                              <th>IP Addressth>
                              <th>Continentth>
                              <th>Countryth>
                              <th>Cityth>
                            tr>
                          thead>
                          <tfoot>
                            <tr>
                              <th>Timeth>
                              <th>IP Addressth>
                              <th>Continentth>
                              <th>Countryth>
                              <th>Cityth>
                            tr>
                          tfoot>
                          <tbody id="customer-table">
                            {% for data in dataSaved.object_list %}
                            <tr>
                                <td>{{ data.datetime }}td>            
                                <td>{{ data.ip }}td>
                                <td>{{ data.continent }}td>
                                <td>{{ data.country }}td>
                                <td>{{ data.city }}td>
                            tr>
                            {% endfor %}
                          tbody>
                        table>
                      div>
                      <br>
                      <nav aria-label="...">
                        {% if dataSaved.has_other_pages %}
                        <ul class="pagination justify-content-center">
                            {% if dataSaved.has_previous %}
                          <li class="page-item">
                            <a class="page-link" href="?page={{ dataSaved.previous_page_number }}" tabindex="-1">Previousa>
                          li>
                          {% endif %}

                          {% for i in dataSaved.paginator.page_range %}
                             {% if dataSaved.number == i %}
                          <li class="page-item active">
                              <a class="page-link" href="#">{{ i }}<span class="sr-only">(current)span>a>
                           li>
                          {% endif %}
                          {% endfor %} 
                          <li class="page-item">
                            {% if dataSaved.has_next %}
                            <a class="page-link" href="?page={{ dataSaved.next_page_number }}">Nexta>
                            {% endif %}
                          li>
                        ul>
                        {% endif %}
                    nav>
                    div>
                    <div class="card-footer small text-muted">Updated at <span id="session-update-time">{{ now }}span>div>
                  div>
                div>

      div>

    main>

    <footer class="text-muted">
      <div class="container">
        <p class="float-right">
          <a href="#">Back to topa>
        p>
        <p>Need help? <a href="#">Contact site developera> or the <a href="#">site admina>.p>
      div>
    footer>

    
    
    <script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous">script>
    <script>window.jQuery || document.write('<script src="../../assets/js/vendor/jquery-slim.min.js"><\/script>')script>
    <script src="https://getbootstrap.com/docs/4.0/assets/js/vendor/popper.min.js">script>
    <script src="https://getbootstrap.com/docs/4.0/dist/js/bootstrap.min.js">script>
    <script src="https://getbootstrap.com/docs/4.0/assets/js/vendor/holder.min.js">script>
  body>
html>

On the monitor/templates/home.html file:

html>
<html lang="en">
  <head>
    
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous">

    <title>Hello, world!title>
  head>
  <body>
    <h1>Hello, world!h1>

    

    
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" crossorigin="anonymous">script>

    
    
  body>
html>

Testing if it Works!

Migrate, then run the server by running the commands below then navigating to the respective port:

(myenv) $ python manage.py makemigrations
(myenv) $ python manage.py migrate
(myenv) $ python manage.py runserver

Below is a snapshot of my implementation:

Thanks for reading!

I hope you enjoyed this blog as much as i did, incase of any question, feel free to ask on the comment section below. Note, you can play with this data as you like, like adding a date range filter to show visitors by date or showing the visitors by country.

Deployment Web Security Checklist | Secure your Django App and Apache Server

Paul Kiragu — Fri, 24 Sep 2021 13:58:39 GMT

Hello DEV World!

Ensuring that your website or open web application is secure is critical. Even simple bugs in your code can result in private information being leaked, and people are out there trying to find ways to steal data. In this article, we are going to check Django and Apache security vulnerabilities and how to fix them.

Run manage.py check --deploy

Some of the checks we are going to cover can be automated using the command below so lets start by running the command:

python manage.py check --deploy

You should see some descriptions which provide information about your Django web application vulnerabilities. Note that these may not be the only vulnerabilities in your system. Here are my checklist warnings:

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W009) Your SECRET_KEY has less than 50 characters, less than 5 unique characters, or it's prefixed with 'django-insecure-' indicating that it was generated automatically by Django. Please generate a long and random SECRET_KEY, otherwise many of Django's security-critical features will be vulnerable to attack.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE, but you have not set CSRF_COOKIE_SECURE to True. Using a secure-only CSRF cookie makes it more difficult for network traffic sniffers to steal the CSRF token.
?: (security.W018) You should not have DEBUG set to True in deployment.
?: (security.W020) ALLOWED_HOSTS must not be empty in deployment.

Indepth Web Scanning using Mozilla Observatory

Mozilla Observatory is a tool that is geared towards informing website owners of best practices for securing their sites, covering everything from personal blogs to eCommerce. The tool uses a scoring system to determine how vulnerable or how well implemented security is on your website. The site also includes third-party scanners which test other security aspects of your site.

Here is a sample scan:

Checklist 1: HTTP Strict Transport Security

If a website accepts a connection through HTTP and redirects to HTTPS, users may initially communicate with the non-encrypted version of the site before being redirected. This creates an opportunity for a man-in-the-middle attack by allowing a redirect to a malicious site instead of the secure version of the original site.

Add the code below to your settings.py file to inform the browser that it should never load a site using HTTP and should automatically convert all attempts to access the site using HTTP to HTTPS requests instead.

SECURE_HSTS_SECONDS = 15780000  # 6 Months as Recommended
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True

Checklist 2: X-XSS-Protection

Cross-site scripting (XSS) is a security exploit which allows an attacker to inject into a website malicious client-side code. This code is executed by the victims and lets the attackers bypass access controls and impersonate users.

To block pages from loading when they detect reflected XSS attacks, make sure django.middleware.security.SecurityMiddleware is present in middleware's list and add following lines in your settings.py:

SECURE_BROWSER_XSS_FILTER = True
SECURE_CONTENT_TYPE_NOSNIFF = True

Checklist 3: CSRF

CSRF (Cross-Site Request Forgery) is an attack that impersonates a trusted user and sends a website unwanted commands. CSRF attacks typically attempt to change server state, but can also be used to gain access to sensitive data. This can be done, for example, by including malicious parameters in a URL behind a link that purports to go somewhere else.

Once you’ve set up HTTPS, add these lines in your settings.py:

CSRF_COOKIE_SECURE = True #to avoid transmitting the CSRF cookie over HTTP accidentally.
SESSION_COOKIE_SECURE = True #to avoid transmitting the session cookie over HTTP accidentally.

Checklist 4: Redirect HTTP to HTTPS

Once you have SSL installed, you need to redirect visitors to HTTPS site. Add following line to your settings.py to force Django redirect all non-HTTPS requests to HTTPS.

SECURE_SSL_REDIRECT = True

Checklist 5: Content Security Policy(CSP)

Here is where things get interesting. Content Security Policy (CSP) is an added layer of security that helps to detect and mitigate certain types of attacks, including Cross-Site Scripting (XSS) and data injection attacks. These attacks are used for everything from data theft to site defacement to distribute malware. Having third party code creates the need of configuring CSP. This may include inline scripts and styles.

Django has an inbuilt module to implement csp headers.

First, install django-csp via pip or from source:

pip install django-csp

From source:

git clone https://github.com/mozilla/django-csp.git
cd django-csp
python setup.py install

Now edit your project’s settings module, to add the django-csp middleware to MIDDLEWARE, like so:

MIDDLEWARE = (
    # ...
    'csp.middleware.CSPMiddleware',
    # ...
)

Note: Middleware order does not matter unless you have other middleware modifying the CSP header. That should do it! Lets go on to configuring CSP by adding following lines to your settings.py

# Content Security Policy
CSP_DEFAULT_SRC = ("'none'", )
CSP_BASE_URI = ("'none'", )
CSP_FRAME_ANCESTORS = ("'none'", )
CSP_FORM_ACTION = ("'self'", )
CSP_STYLE_SRC = ("'self'", 'cdnjs.cloudflare.com','fonts.googleapis.com', 'stackpath.bootstrapcdn.com', 'getbootstrap.com')
CSP_SCRIPT_SRC = ("'self'", 'ajax.googleapis.com', 'stackpath.bootstrapcdn.com', 'code.jquery.com', 'unpkg.com', 'cdnjs.cloudflare.com', 'code.jquery.com', 'getbootstrap.com')
CSP_IMG_SRC = ("'self'", 'cdn0.iconfinder.com', 'media.giphy.com','res.cloudinary.com')
CSP_FONT_SRC = ("'self'",'cdnjs.cloudflare.com','fonts.googleapis.com', 'fonts.gstatic.com')
CSP_INCLUDE_NONCE_IN = ("script-src", "style-src")

Its really important to clean your code from all these inline styles and scripts. However, some external resources such as the ones included above may be allowed in your CSP policy. Make sure to look at the django-csp documentation in order to know what to update above as the sample above is an example of common tags loaded on the HTML page and also includes inline scripts and style. If you wish to load your scripts from source only then modify the code above to:

# Content Security Policy
CSP_DEFAULT_SRC = ("'none'", )
CSP_BASE_URI = ("'none'", )
CSP_FRAME_ANCESTORS = ("'none'", )
CSP_FORM_ACTION = ("'self'", )
CSP_STYLE_SRC = ("'self'", )
CSP_SCRIPT_SRC = ("'self'", )
CSP_IMG_SRC = ("'self'", )
CSP_FONT_SRC = ("'self'", )

Checklist 6: Critical settings

SECRET_KEY

The secret key must be a large random value and it must be kept secret.

Make sure that the key used in production isn’t used anywhere else and avoid committing it to source control. This reduces the number of vectors from which an attacker may acquire the key.

Instead of hardcoding the secret key in your settings module, consider loading it from an environment variable:

import os
SECRET_KEY = os.environ['SECRET_KEY']

Or from a file:

with open('/etc/secret_key.txt') as f:
    SECRET_KEY = f.read().strip()

DEBUG

You must never enable debug in production. Change the following lines in your settings.py file to False to disable it:

DEBUG = False

Checklist 6: Environment Specific settings

ALLOWED_HOSTS

When we set DEBUG = False above, Django doesn’t work at all without a suitable value for ALLOWED_HOSTS. This setting is required to protect your site against some CSRF attacks. Change the following line on the settings.py file to point this to your website's domain.

ALLOWED_HOSTS = ['alienx.tech'] #should point to your domain name

DATABASES

Database connection parameters are probably different in development and in production. Database passwords are very sensitive. You should protect them exactly like we did with SECRET_KEY above. For maximum security, make sure database servers only accept connections from your application servers.

Sample End Result Ranking by Mozilla Observatory

Here is my website which implements the features above:

Please note that this does not mean that our site is proof to hacking by getting an A+, this just means that our site is keeping up with the latest web security standards as per the database of vulnerabilities given by the Mozilla Observatory API. Lets us now proceed to securing our Apache Webserver. Please proceed with the commands below if you are well familiar with apache

How to Secure Your Apache Server

Apache is a popular, open-source web server available for both Linux and Windows systems. It allows configuration for a diverse range of use cases, from HTML webpages to HyperText Preprocessor (PHP) dynamic web application content.Apache provides a secure and robust platform to deploy your web applications. However, it is still important to install the latest security patches and configure the server properly to establish a secure environment for your web applications.

Non-Privileged User Account

A non-privileged user is a user that does not belong to the Dynamic Data Masking administration group. The purpose of having one is to restrict the user from unnecessary access to certain tasks within a system. In the context of an Apache web server, this means that it should work in a restricted environment with only the necessary permissions. By default, Apache runs with daemon account privileges. You can create a separate non-root user account to avoid threats in case of security attacks.

Keep Apache Updated

Apache is famous for providing a secure platform with a highly concerned developer community that rarely faces any security bugs. The good thing about being open source is that for every bad guy out there, there is always around 10 who are good and looking for this bugs to patch. Nevertheless, it is normal to discover issues once the software is released. Hence, it is essential to keep the web server up to date to avail the latest security features.

Run the command below:

sudo apt-get update
sudo apt-get upgrade

Disable Server Signature

A server signature is the public identity of your web server and contains sensitive information that could be used to exploit any known vulnerability for example your apache version. The default configuration of an Apache Server exposes a lot of details about the server and its settings.

You can disable these directives by editing the apache2.conf file via vim/nano and add the following directive:

...snip...
ServerSignature Off
...snip...
ServerTokens Prod
...snip...

Disable Server Directory Listings

The Directory listings displays all content saved in the root folder or sub-directories. The directory files can include sensitive information not intended for public display, such as PHP scripts, configuration files, files containing passwords, logs, etc. This directive can also be added in the .htaaccess of your main website directory To disallow directory listings, change the Apache server configuration file by editing the apache2.conf file as:

...snip...

<Directory /var/www>

Options -Indexes

Directory>

...snip...

.htaccess file Settings

htaccess files (or "distributed configuration files") provide a way to make configuration changes on a per-directory basis. A file, containing one or more configuration directives, is placed in a particular document directory, and the directives apply to that directory, and all subdirectories thereof. In cases where a user can upload files to the server, this can be exploited by an attacker to upload his or her own “.htaccess” file with malicious configurations. So, if you are not using this feature, you can disable the .htaccess directive by editing the apache.conf file like so:

...snip...
#AccessFileName .htaccess
...snip...

Prevent Slow Loris and DoS Attack

The default installation of an Apache server forces it to wait for requests from clients for too long, which subjects the server to Slow Loris and DoS attacks. The apache2.conf configuration file provides a directive that you can use to lower the timeout value to a few seconds to prevent these types of attacks by editing your apache2.conf file like so:

Timeout 60

Disable Unnecessary HTTP Requests

Unlimited HTTP/HTTPS requests can also lead to low server performance or a DoS attack. You can limit receiving HTTP requests per-directory by using LimitRequestBody to less than 100K. For instance to create a directive for a folder, add the LimitRequestBody directive below AllowOverride All:

...snip...

<Directory /var/www/your_website>

Options -Indexes

AllowOverride All

LimitRequestBody 995367

Directory>

...snip...

Thanks for tuning in!

Thats it for today folks. In the meantime, there are plenty of other ways (not listed above) to secure your Apache web server and Django Web Application, as well. Continue researching and keeping yourself updated about new directives and modules to secure your server further and keep on cross checking your website against the Mozilla observatory scan

Automate Exam Research with Django, Nanonets and Google Search API

Paul Kiragu — Sun, 19 Sep 2021 17:57:38 GMT

Welcome to the first post of my blog. In this article, we shall be building a Django Web Application that allows a user to take a photo of an examination paper, and the application automatically Google's everything for the user and gives back the answers with their descriptions and links of the sources. Before I start, I would like to say that I did not write this piece of code for the sake of cheating in examination or giving students a platform for cheating, this tool is simply for making examination revision easier and for educational purposes only. Cheating in examinations is illegal and would probably get you expelled!

The full project can be found on my GitHub repository and the demo MAY be found here

Prerequisites

We will be using Python3.6+, Django web framework, Nanonets for character extraction from an image, Cloudinary for image storage and Google Search API for performing the searches.

Following Python best practices, we will create a virtual environment for our project, and install the required packages.

First, create the project directory.

$ mkdir djangoapp
$ cd djangoapp

Now, create a virtual environment and install the required packages.

For macOS and Unix systems:

$ python3 -m venv myenv
$ source myenv/bin/activate
(myenv) $ pip install django requests cloudinary

For Windows:

$ python3 -m venv myenv
$ myenv\Scripts\activate
(myenv) $ pip install django requests cloudinary

Configuring Cloudinary

You will need a Cloudinary account which we shall be using in this project for storing images uploaded by users to be scanned. Cloudinary is an end-to-end image- and video management solution for websites and mobile apps, covering everything from image and video uploads, storage, manipulations, optimizations to delivery. Once you sign up or login successfully, head over to your Cloudinary console and click on start configuring to get your API key. Note this carefully as we shall be using this shortly

Nanonets for text extraction

Nanonets is a platform that allows for the capture of data from documents instantly at the same time reducing turnaround times and the manual effort required. You can upload your own data and train a model, acquire the JSON responses of each prediction to integrate it with your own systems, and build machine learning-powered apps built on state-of-the-art algorithms and a strong infrastructure. Nanonets also comes with a free tier which allows you to play around with the data on small scale. Follow the steps below to get started building the model:

Get your free API Key

Get your free API Key from http://app.nanonets.com/#/keys

Upload Images For Training

On the nanonets console, click on new model then create your own, to build your own extractor. Then click upload to add the images to be used for training. You can find them here. Remember, the more the train data the better the results. The minimum required is 10 files, the recommended is 50 - 100.

Once the images have been successfully uploaded, click on next to start managing the model labels. Incase of any errors, the system will not proceed to the manage label section so be sure to ensure you follow the criteria of upload stated. Add one label and then click on start training. This label will be what we are extracting from the question paper.

The model learns from the examples before it can be trained, Mark the data for the label question we created above by simply drawing over the question on the question papers uploaded. Do this on the images till there are at least 10 examples per label. The more the examples, the better the results.

Once you are done, start training the model by clicking that button. Once the model training finishes, you will be notified by email and the model state will change on the console to green meaning the model is ready. Depending on the number of images you used, this process will typically take a couple of minutes.

Setting Up Your Django Application

Before we start integrating Django with the model we just trained on nanonets, it is worth stating that we do not need Django to integrate our model, you can use any language/Framework you wish to do the integration. The nanonets console even has some code samples on various language integrations.

First, navigate to the directory djangoapp we created and establish a Django project.

(myenv) $ django-admin startproject mainapp

This will auto-generate some files for your project skeleton:

mainapp/
    manage.py
    mainapp/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Now, navigate to the directory you just created (make sure you are in the same directory as manage.py) and create your app directory.

(myenv) $ python manage.py startapp exam

This will create the following:

exam/
    __init__.py
    admin.py
    apps.py
    migrations/
        __init__.py
    models.py
    tests.py
    views.py

On the mainapp/settings.py file, look for the following line and add the app we just created above.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'exam', #new line
]

Ensure you are in the exam directory then create a new directory called templates and a new file called urls.py. Your directory structure of exam application should look like this

exam/
    __init__.py
    admin.py
    apps.py
    migrations/
    templates/
        __init__.py
    models.py
    tests.py
    urls.py
    views.py

Ensure your mainapp/urls.py file, add our exam app URL to include the URLs we shall create next on the exam app:

"""mainapp URL Configuration

The `urlpatterns` list routes URLs to views. For more information please see:
    https://docs.djangoproject.com/en/3.2/topics/http/urls/
Examples:
Function views
    1. Add an import:  from my_app import views
    2. Add a URL to urlpatterns:  path('', views.home, name='home')
Class-based views
    1. Add an import:  from other_app.views import Home
    2. Add a URL to urlpatterns:  path('', Home.as_view(), name='home')
Including another URLconf
    1. Import the include() function: from django.urls import include, path
    2. Add a URL to urlpatterns:  path('blog/', include('blog.urls'))
"""
from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path(' ', include('exam.urls')),#exam app url
]

Now, on the exam/urls.py file, add our website there:

from django.urls import path, include
from . import views

urlpatterns = [
    path(' ', views.exams, name="exams"),#exam researcher view to be created next
]

On the mainapp/settings.py file add the configuration key available on the Cloudinary Dashboard:

cloudinary.config( 
  cloud_name = "REPLACE WITH YOUR KEY", 
  api_key = "AND REMEMEBER", 
  api_secret = "NEVER EXPOSE KEYS DURING PRODUCTION" 
)

On the exam/views.py, add the following lines of code: Explanation available through comments

from django.shortcuts import render, redirect
import requests
import cloudinary.uploader
from concurrent.futures import ThreadPoolExecutor, as_completed
from time import time

#exams researcher
def exams(request):
    if request.method == 'POST':
        files = request.FILES['files']
        upload_data = cloudinary.uploader.upload(files)
        imageData = upload_data['secure_url']
        nanoNetsApi = 'https://app.nanonets.com/api/v2/OCR/Model/418edf37-dsdadfgs/LabelUrls/' #replace THE MODEL ID 418edf37-dsdadfgs WITH YOURS found on the nanonets model console https://app.nanonets.com/#/models
        headers = {'accept': 'application/x-www-form-urlencoded'}
        data = {'urls' : [imageData]}
        response = requests.request('POST', nanoNetsApi, headers=headers, auth=requests.auth.HTTPBasicAuth('REPLACE', ''), data=data)
        test = response.json()
        for quiz in test['result']:
            predicted = quiz['prediction']
            q1 = predicted[0]['ocr_text']
            q2 = predicted[1]['ocr_text']
            q3 = predicted[2]['ocr_text']
            q4 = predicted[3]['ocr_text']
            q5 = predicted[4]['ocr_text']
            q6 = predicted[5]['ocr_text']
            q7 = predicted[6]['ocr_text']
            q8 = predicted[7]['ocr_text']
            q9 = predicted[8]['ocr_text']
            q10 = predicted[9]['ocr_text']
            q11 = predicted[10]['ocr_text']
            q12 = predicted[11]['ocr_text']
            q13 = predicted[12]['ocr_text']
            q14 = predicted[13]['ocr_text']

        q1Format = q1.replace(" ", "+")
        q2Format = q2.replace(" ", "+")
        q3Format = q3.replace(" ", "+")
        q4Format = q4.replace(" ", "+")
        q5Format = q5.replace(" ", "+")
        q6Format = q6.replace(" ", "+")
        q7Format = q7.replace(" ", "+")
        q8Format = q8.replace(" ", "+")
        q9Format = q9.replace(" ", "+")
        q10Format = q10.replace(" ", "+")
        q11Format = q11.replace(" ", "+")
        q12Format = q12.replace(" ", "+")
        q13Format = q13.replace(" ", "+")
        q14Format = q14.replace(" ", "+")

        q1SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q1Format
        q2SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q2Format
        q3SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q3Format
        q4SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q4Format
        q5SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q5Format
        q6SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q6Format
        q7SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q7Format
        q8SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q8Format
        q9SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q9Format
        q10SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q10Format
        q11SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q11Format
        q12SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q12Format
        q13SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q13Format
        q14SearchUrl = "https://google-search3.p.rapidapi.com/api/v1/search/q="+q14Format

        searchHeader = { 'x-user-agent': "desktop", 'x-rapidapi-host': "google-search3.p.rapidapi.com", 'x-rapidapi-key': "43628cd680msh1812b1660500eb7p182976jsn5dda2f77f08f" }

        searchResponseQ1 = requests.request("GET", q1SearchUrl, headers=searchHeader)
        searchResponseQ2 = requests.request("GET", q2SearchUrl, headers=searchHeader)
        searchResponseQ3 = requests.request("GET", q3SearchUrl, headers=searchHeader)
        searchResponseQ4 = requests.request("GET", q4SearchUrl, headers=searchHeader)
        searchResponseQ5 = requests.request("GET", q5SearchUrl, headers=searchHeader)
        searchResponseQ6 = requests.request("GET", q6SearchUrl, headers=searchHeader)
        searchResponseQ7 = requests.request("GET", q7SearchUrl, headers=searchHeader)
        searchResponseQ8 = requests.request("GET", q8SearchUrl, headers=searchHeader)
        searchResponseQ9 = requests.request("GET", q9SearchUrl, headers=searchHeader)
        searchResponseQ10 = requests.request("GET", q10SearchUrl, headers=searchHeader)
        searchResponseQ11 = requests.request("GET", q11SearchUrl, headers=searchHeader)
        searchResponseQ12 = requests.request("GET", q12SearchUrl, headers=searchHeader)
        searchResponseQ13 = requests.request("GET", q13SearchUrl, headers=searchHeader)
        searchResponseQ14 = requests.request("GET", q14SearchUrl, headers=searchHeader)

        q1resp = searchResponseQ1.json()
        q1res = q1resp.get('results')

        q2resp = searchResponseQ1.json()
        q2res = q2resp.get('results')

        q4resp = searchResponseQ4.json()
        q4res = q4resp.get('results')

        q5resp = searchResponseQ5.json()
        q5res = q5resp.get('results')

        q6resp = searchResponseQ6.json()
        q6res = q6resp.get('results')

        q7resp = searchResponseQ7.json()
        q7res = q7resp.get('results')

        q8resp = searchResponseQ8.json()
        q8res = q8resp.get('results')

        q9resp = searchResponseQ9.json()
        q9res = q9resp.get('results')

        q10resp = searchResponseQ10.json()
        q10res = q10resp.get('results')

        q11resp = searchResponseQ11.json()
        q11res = q11resp.get('results')

        q12resp = searchResponseQ12.json()
        q12res = q12resp.get('results')

        q13resp = searchResponseQ13.json()
        q13res = q13resp.get('results')

        q14resp = searchResponseQ14.json()
        q14res = q14resp.get('results')

        tester = searchResponseQ3.json()
        testa = tester.get('results')

        context = {
            "tester":testa,
            "q1res":q1res,
            "q2res":q2res,
            "q4res":q4res,
            "q5res":q5res,
            "q6res":q6res,
            "q7res":q7res,
            "q8res":q8res,
            "q9res":q9res,
            "q10res":q10res,
            "q11res":q11res,
            "q12res":q12res,
            "q13res":q13res,
            "q14res":q14res,
        }
        return render(request, 'exam_results.html', context)
    return render(request, 'exams.html')

This view function starts by accepting POST data from the exam.html file that we shall create below. The file uploaded by the user is then uploaded to Cloudinary. If successful, the URL of the image generated by cloudinary is then taken and run against nanonets model we created.

Next, on the mainapp/exam/templates directory, create a exams.html and exam_results.html files and paste the code below to render our output

mainapp/exam/templates/exams.html

html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <meta name="description" content="">
    <meta name="author" content="">
    
<title>AlienX - Exam Researchertitle>
    
    <link href="https://getbootstrap.com/docs/4.0/dist/css/bootstrap.min.css" rel="stylesheet">
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css">
    <style>
      #upload{opacity:0}#upload-label{position:absolute;top:50%;left:1rem;transform:translateY(-50%)}.image-area{border:2px dashed rgba(255,255,255,.7);padding:1rem;position:relative}.image-area::before{content:'Uploaded image result';color:#fff;font-weight:700;text-transform:uppercase;position:absolute;top:50%;left:50%;transform:translate(-50%,-50%);font-size:.8rem;z-index:1}.image-area img{z-index:2;position:relative}body{min-height:100vh;background-color:#757f9a;background-image:linear-gradient(147deg,#757f9a 0,#d7dde8 100%)}
  style>
  <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js">script>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.3.1/js/bootstrap.bundle.min.js">script>
  <script>
      /*  ==========================================
    SHOW UPLOADED IMAGE
* ========================================== */
function readURL(input) {
    if (input.files && input.files[0]) {
        var reader = new FileReader();

        reader.onload = function (e) {
            $('#imageResult')
                .attr('src', e.target.result);
        };
        reader.readAsDataURL(input.files[0]);
    }
}

$(function () {
    $('#upload').on('change', function () {
        readURL(input);
    });
});

/*  ==========================================
    SHOW UPLOADED IMAGE NAME
* ========================================== */
var input = document.getElementById( 'upload' );
var infoArea = document.getElementById( 'upload-label' );

input.addEventListener( 'change', showFileName );
function showFileName( event ) {
  var input = event.srcElement;
  var fileName = input.files[0].name;
  infoArea.textContent = 'File name: ' + fileName;
}
  script>
head>

  <body class="text-center">
    <div class="container py-5">

        
        <header class="text-white text-center">
            <h1 class="display-4">Question Researcherh1>
            <p class="lead mb-0">Take Image or Upload existing imagep>
            <p class="mb-5 font-weight-light">Powered by
                <a href="https://alienx.tech" class="text-white">
                    <u>alienXconnectionsu>
                a>
            p>
            <img src="https://res.cloudinary.com/mhmd/image/upload/v1564991372/image_pxlho1.svg" alt="" width="150" class="mb-4">
        header>


        <div class="row py-4">
            <div class="col-lg-6 mx-auto">
                <form method = "post" enctype="multipart/form-data">
                    {% csrf_token %}
                
                <div class="input-group mb-3 px-2 py-2 rounded-pill bg-white shadow-sm">
                    <input id="upload" type="file" name="files" onchange="readURL(this);" class="form-control border-0">
                    <label id="upload-label" for="upload" class="font-weight-light text-muted">Choose filelabel>
                    <div class="input-group-append">
                        <label for="upload" class="btn btn-light m-0 rounded-pill px-4"> <i class="fa fa-cloud-upload mr-2 text-muted">i><small class="text-uppercase font-weight-bold text-muted">Choose filesmall>label>
                    div>
                div>

                
                <p class="font-italic text-white text-center">The image uploaded will be rendered inside the box below.p>
                <div class="image-area mt-4"><img id="imageResult" src="#" alt="" class="img-fluid rounded shadow-sm mx-auto d-block">div>
                <button class="btn btn-lg btn-primary btn-block" type="submit">Nextbutton>
                form>
            div>
        div>
    div>
  body>
html>

mainapp/exam/templates/exam_results.html


html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <meta name="description" content="">
    <meta name="author" content="">
    <link rel="icon" href="https://getbootstrap.com/docs/4.0/assets/img/favicons/favicon.ico">

    <title>Top navbar example for Bootstraptitle>


    
    <link href="https://getbootstrap.com/docs/4.0/dist/css/bootstrap.min.css" rel="stylesheet">

    
    <link href="https://getbootstrap.com/docs/4.0/examples/navbar-static/navbar-top.css" rel="stylesheet">
  head>

  <body>



    <main role="main" class="container">
        <h1>Researched Questions with their linksh1>

        {% for aa in q1res %}
      <div class="jumbotron">
        <h3>{{aa.title}}h3>
        <p class="lead">{{aa.description}}p>
        <a class="btn btn-lg btn-primary" href="{{aa.link}}" role="button">Site»a>
      div>
      {% endfor %}

      {% for bb in q2res %}
      <div class="jumbotron">
        <h3>{{bb.title}}h3>
        <p class="lead">{{bb.description}}p>
        <a class="btn btn-lg btn-primary" href="{{bb.link}}" role="button">Site»a>
      div>
      {% endfor %}

      {% for tt in tester %}
      <div class="jumbotron">
        <h3>{{tt.title}}h3>
        <p class="lead">{{tt.description}}p>
        <a class="btn btn-lg btn-primary" href="{{tt.link}}" role="button">Site»a>
      div>
      {% endfor %}

      {% for ee in q4res %}
      <div class="jumbotron">
        <h3>{{ee.title}}h3>
        <p class="lead">{{ee.description}}p>
        <a class="btn btn-lg btn-primary" href="{{ee.link}}" role="button">Site»a>
      div>
      {% endfor %}

      {% for ff in q5res %}
      <div class="jumbotron">
        <h3>{{ff.title}}h3>
        <p class="lead">{{ff.description}}p>
        <a class="btn btn-lg btn-primary" href="{{ff.link}}" role="button">Site»a>
      div>
      {% endfor %}

      {% for gg in q6res %}
      <div class="jumbotron">
        <h3>{{gg.title}}h3>
        <p class="lead">{{gg.description}}p>
        <a class="btn btn-lg btn-primary" href="{{gg.link}}" role="button">Site»a>
      div>
      {% endfor %}

      {% for hh in q7res %}
      <div class="jumbotron">
        <h3>{{hh.title}}h3>
        <p class="lead">{{hh.description}}p>
        <a class="btn btn-lg btn-primary" href="{{hh.link}}" role="button">Site»a>
      div>
      {% endfor %}

      {% for ii in q8res %}
      <div class="jumbotron">
        <h3>{{ii.title}}h3>
        <p class="lead">{{ii.description}}p>
        <a class="btn btn-lg btn-primary" href="{{ii.link}}" role="button">Site»a>
      div>
      {% endfor %}

      {% for jj in q9res %}
      <div class="jumbotron">
        <h3>{{jj.title}}h3>
        <p class="lead">{{jj.description}}p>
        <a class="btn btn-lg btn-primary" href="{{jj.link}}" role="button">Site»a>
      div>
      {% endfor %}


      {% for kk in q10res %}
      <div class="jumbotron">
        <h3>{{kk.title}}h3>
        <p class="lead">{{kk.description}}p>
        <a class="btn btn-lg btn-primary" href="{{kk.link}}" role="button">Site»a>
      div>
      {% endfor %}




    main>

    
    
    <script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous">script>
    <script>window.jQuery || document.write('<script src="https://getbootstrap.com/docs/4.0/assets/js/vendor/jquery-slim.min.js"><\/script>')script>
    <script src="https://getbootstrap.com/docs/4.0/assets/js/vendor/popper.min.js">script>
    <script src="https://getbootstrap.com/docs/4.0/dist/js/bootstrap.min.js">script>
  body>
html>

One major problem with the system, it takes around 40 seconds to run all those requests, this is where concurrency comes in. Try running the files below which simulate the request above and tell the difference! The code was able to run from 40 seconds to 4 seconds just by adding a few lines of code

without_concurrency.py

import requests
from concurrent.futures import ThreadPoolExecutor, as_completed
from time import time

q1Format = "elon+musk"
q2Format = "elon+musk+age"
q3Format = "elon+musk+networth"
q4Format = "elon+musk+wife"
q5Format = "elon+musk+h"
q6Format = "elon+musk+faces"
q7Format = "elon+musk+kenya"
q8Format = "elon+musk+uhuru+kenyatta"
q9Format = "elon+musk+m"
q10Format = "elon+musk+de"
q11Format = "elon+musk+y"
q12Format = "elon+musk+mate"
q13Format = "elon+musk+uk"
q14Format = "elon+musk+london"

url_list = [
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q1Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q2Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q3Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q4Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q5Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q6Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q7Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q8Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q9Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q10Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q11Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q12Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q13Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q14Format,
]

searchHeader = { 'x-user-agent': "desktop", 'x-rapidapi-host': "google-search3.p.rapidapi.com", 'x-rapidapi-key': "43628cd680msh1812b1660500eb7p182976jsn5dda2f77f08f" }


def download_file(url):
    html = requests.get(url, headers=searchHeader, stream=True)
    return html.status_code

start = time()

for url in url_list:
    print(download_file(url))

print(f'Time taken: {time() - start}')

with_concurrency.py

import requests
from concurrent.futures import ThreadPoolExecutor, as_completed
from time import time

q1Format = "elon+musk"
q2Format = "elon+musk+age"
q3Format = "elon+musk+networth"
q4Format = "elon+musk+wife"
q5Format = "elon+musk+salary"
q6Format = "elon+musk+faces"
q7Format = "elon+musk+kenya"
q8Format = "elon+musk+uhuru+kenyatta"
q9Format = "elon+musk+trump"
q10Format = "elon+musk+spacex"
q11Format = "elon+musk+cats"
q12Format = "elon+musk+mate"
q13Format = "elon+musk+uk"
q14Format = "elon+musk+london"

url_list = [
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q1Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q2Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q3Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q4Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q5Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q6Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q7Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q8Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q9Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q10Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q11Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q12Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q13Format,
    "https://google-search3.p.rapidapi.com/api/v1/search/q="+q14Format,
]


searchHeader = { 'x-user-agent': "desktop", 'x-rapidapi-host': "google-search3.p.rapidapi.com", 'x-rapidapi-key': "43628cd680msh1812b1660500eb7p182976jsn5dda2f77f08f" }


def download_file(url):
    global html
    html = requests.get(url, headers=searchHeader, stream=True)
    return html.status_code

start = time()

processes = []
with ThreadPoolExecutor(max_workers=20) as executor:
    for url in url_list:
        processes.append(executor.submit(download_file, url))

for task in as_completed(processes):
    print(task.result())


print(f'Time taken: {time() - start}')

Testing It Works

Make sure you are in the directory with manage.py, and run

(myenv) $ python manage.py runserver

You should see the port that your Django application is running on. But this is still running from your computer. Check out the snapshots below on the final product

Thanks for Reading!

This is my first blog so I know might have left out a lot of details especially on the views part, incase of any question, let me know in the comment section.