Blog

Software development without LLMs: A methodology and a reasoning

As experiences gather up and technology improves, there are more and more evaluations and strategies coming in - all about when and why to use this data-center-driven LLM technology. A lot of decision-makers have already jumped aboard the AI/LLM train. Maybe you're one of them - and I kind of hope so, because this blog post is for you!

My departing point for everything here is this: For the sake of not burning down the planet, it seems pretty decent to try to argue in favor of NOT needing LLM-assistance for producing software projects, whether sloppy ones or state-of-the-art.

I find it more interesting to ask questions like:

How can we improve software development without resorting to cloud-based LLM?
How can we make software development have less direct impact on the climate?
How can we maintain software over a longer period of time?
How can we write more effective test cases?

The answers to such questions are LONG, often branching deep into methodology / architecture / programming language.

I.e. I'm not asking "how can we write more code?" or "how can people without knowledge of software development be empowered to write software?" - to me they are bizarre questions. Half the planet's population can drive a car, so if self-driving cars are the answer, then what exactly was the question?

And I keep returning to this main finding: We didn't have any real problems with producing software, so the AI hype is trying to solve a non-problem. If you think about it for a moment, things were going pretty well in the software industry and people were mostly concerned with things like: Monopolies and vendor lock-in, privacy/data issues, e-waste, infosec etc. For software development itself, we had nice conferences, nice colleagues, we knew how to do our work pretty well.

In cases like infosec, then we have LLM technology solving issues that it's producing itself. Got problems caused by AI? Use AI to solve them!

Let's first break down some major areas of applying LLM to software, and then finally define a methodology that doesn't actually use LLMs but instead focuses on the benefits that arise from NOT using LLMs (aside from the omnipresent benefits like not burning down the planet).

LLM reviews
LLM security scanning
LLM research
LLM code generation

Making decisions in teams

It's easy to feel joy of building and crafting your own software. Seeing the results, knowing that "I did it". Maybe someone else will notice, too, and tell you about the great work that you did.

We all have agency and motivation - and we're all equals and equally different. Sure, some are more experienced than others and will say something correct more often. Someone will have more power, and others will listen to them more than others. That's not exactly equal, so what do I mean with everyone being equals? Well what I mean is that everyone should at least be offered the same chances and resources to achieve the same goals. That goes for software teams as well.

Everyone should be allowed to fail and make amends, say something incorrect and change their mind.

A primer on Django app design

In this follow-up to A primer on Django project structure, we're gonna look at the layout of individual Django apps.

In the former topic, I went as far as to define a basic, universal project layout and also intended to fuel a discussion about templates, tutorials and the recent django new command. But this blog post is more an introduction to operating your Django project once you start filling it in with Django apps, Python modules and Python packages.

When things go wrong 🙀

Let's look at some examples of a layout with symptoms of needing a refactor...

Circular imports

Have you ever seen local imports (below example) run wild? The local imports are there to avoid the top-level imports becoming circular. But they often just keep increasing in numbers, as they're a sign that the dependency coupling between Django apps or the modules inside the app is wrong. When you see a single local import in an entire project, it's maybe just fine. But once you see the first one, it might also be a sign that the code isn't optimally structured. A so-called smell.

def do_something(...)
    # Use local import to avoid circularity
    # (because other_app.models depends on this module)
    from other_app.models import MyModel
    ...

Confusion about where to add new stuff

Is having some simple answers to what goes where is becoming problematic? It's not nice to be in situations where modules overlap in scope/domain and don't provide clear guidance.

In doubt about where your functionality should go? Add a new app / module! 😖 🙃

Lack of motivation

If you keep throwing functionalities into a messy Django project, it will eventually become a mess -- and that might quickly get worse when people have a perception that it needs to be rewritten, anyways. "Someone in the future will have to deal with this" is the anti-thesis of well-structured and documented code because that "someone" is usually just yourself 6-12 months from now.

Memory feels fuzzy

Likely the hardest symptom to grasp is when you start feeling that you're having a hard time quickly remembering which file to open in your project. We have IDE auto-completion for most things that can be imported, but even that can become difficult if naming conventions differ across your project.

App and module domains: Planning and definition

There's lots of truth in the Unix Philosophy: Do one thing and do it well. But we need to somehow translate this into how we organize our Apps and their modules inside our Django Project.

In order to dive into architecture, you can watch Paul Wolf's Clean Architecture with Django Rethinking basic assumptions from DjangoCon Europe 2021.

Let's consider the point that the volatile parts of your code should depend on the more stable parts:

admin.py: should be rapidly adapting to your day-to-day needs. Consider these definitions volatile.
views.py: with all their UX logic: Very volatile!
urls.py: is almost like a kind of configuration: You can change a URL and everything else should just work because we have urls.reverse() etc - of course that's not to say that we don't have permalinks and external references.. so we probably wanna consider it a more stable area.
...
models.py: Requires migrations for many changes and can be considered one of your app's most "stable" interfaces. Once you've added a method to a model, you'll start to depend on it in other modules, and so these changes should be seen as "stable".

So all-in-all, the order of your module dependencies here reflect that general idea.

Recipe: Layout of each app

Let's try to answer these questions about stability, volatility and dependencies. We'll use common conventions (that already took this into account!),, and we end up with a blueprint of modules that a Django app contains and what they're for.

Here's the example recipe of a Django app, with some inevitable conventions for the module names:

src/
  apps/
    my_app/
      admin.py
      forms.py
      models.py
      shortcuts.py
      tasks.py
      views.py
      urls.py
      utils.py

We'll step through these choices - it might sound like defining abilities of chess pieces, but it's different since each module is strictly higher in ranking than its successors in the list.

urls.py sits on top of everything else - it MUST define url_patterns and isn't imported by much else than Django's URL resolver, but that's also by design, because actually this is a very volatile piece of code: You want to be able to constantly add and refactor URLs and what they point to.
views.py comes next, the only module that imports from views is: urls. But views import pretty much everything else.
admin.py can use tasks, shortcuts and models - this module sits almost at the top of the "food chain" so to say. But I would recommend to avoid depending on views and keep all admin views local to the admin module, and even better: As methods of your ModelAdmins.
shortcuts.py can import from all other modules, except your views which normally consume the shortcuts. Don't confuse shortcuts with a service layer, but use it to avoid having one.
forms.py contains for instance all the ModelForms used by your views and depends on your models. It will also often spawn tasks since it processes form data.
tasks.py comes next - it contains all your async tasks and can depend on models and acts as a kind of controller/service layer.
models.py is second-to-last: If you have more modules, make sure to make them depend on models, not the other way around.
utils.py is a utility library that ALL OTHER modules can depend on. If my_app.utils depends on other things in my_app, then the utilities themselves have utilities inside my_app.. that doesn't work! Under no circumstances let your app's utility layer end up depending on your app's models etc.: What you're looking for is model methods, managers etc. Utilities are for instance functions that call Django itself, convert a text string, localizes a number etc. A good rule of thumb is to never access the database from utils.py.

The thinking behind what utils and shortcuts should scope to and depend on is from Django itself, i.e. in Django, the utils module has close to zero dependencies to other Django modules.

Conventions for templates and static media

I also just wanna mention this, since it's overlooked once too often: Remember to use sub-folders as namespaces for your templates and static files since they are subject to various kinds of "overrides" in how templates and static media is resolved and collected.

So let's say you have an app foobar, then it should have this structure:

src/
  apps/
      foobar/
          templates/
              foobar/  # <- that's the important sub-folder!
                  base.html
            static/
              foobar/  # <- and this!
                  menu.js

Nice discussions / exercise

Here is some homework/discussion:

Can you guess where widgets.py would fit in?
Can you guess where exceptions.py would fit in? And would you want one?
Should we have a fields.py module?
What code should execute at load time?
How does tests reflect on stable vs. volatile?
Are Django template files stable or volatile pieces of code?

Tips

Test-ability wins

If you think about writing a unit test, then it's always easier to write if your methods and functions are defined by simple input-output logic.

This also wins in the longer run: If you are servicing a large project with lots of data, you can for instance need to manage data directly in the shell. When things are broken and new exceptions show up in your monitoring, you might need to improvise patching up your data. Did the order processing break and you want to call order.complete()? That works nicely if you've defined it as a method on a model, but not if the order logic is nested deep within a view that relies on an HttpRequest object. That same logic is true for unit testing. And it'll make everything more readable.

Refactoring code into smaller routines isn't "premature abstraction": This happens when you invent functionality and data structures while refactoring into smaller routines. See the AHA principle.

Skip `managers.py` - put query logic in the models

A Django model should declare its managers, it's great to have custom managers and business logic reusable and manifested in managers. And it's great for unit testing! But the managers usually depend on the models. So at this stage, they're mutually dependent and should live in the same module.

You can also use nested classes, since a manager is rarely reused outside of its model. It looks like this:

class MyModel(models.Model):
    class DefaultQuerySet(models.QuerySet):
        def active(self):
            return self.filter(is_active=True)

    objects = DefaultQuerySet.as_manager()

But then models.py will become very long? YES! And that's not really a problem since IDEs help you navigate code structures. Having convoluted import patters is much worse, and declaring custom QuerySets inside the model is very easy to regard as Locality of Behavior.

Do have a custom user model

It's awful to change from using the built-in User model from django.contrib.auth, so I recommend to always bootstrap your project with a users app. See Django Docs about customizing the User model.

Avoid the Service layer

You could probably put a service layer somewhere next to shortcuts. I would recommend to avoid this, though, since most of the time it will simply do things that should otherwise have been encoded in views, tasks, model methods and manager/queryset methods.

If you think services are the answer to encapsulating business logic (as they're usually lauded for), you can read this guide:

https://github.com/HackSoftware/Django-Styleguide?tab=readme-ov-file#services

My own short take is that business logic should be implemented in APIs, so if you're doing API-driven stuff, put your business logic in API views, serializers and model methods/managers/querysets. If you're doing a full-stack Django project, then the business logic resides in views and model methods/managers/querysets. This will make your life simpler and your applications will contain a better structure and better tests.

BUT if your service layer is actually going to be very big and it needs to live its own life, you should probably read the HackSoft guide and take their word:

Sometimes, the service layer is not always looking like a single file with a bunch of functions, but rather like an entire Python module with well-defined interface, so we can encapsulate something complex & isolate it from the rest of the app.

We’ve even gone further creating a layer of types, between the ORM & the business logic (this is an overkill for most cases, but there are cases where it helps)

The general idea - separation of concerns - is key.

https://forum.djangoproject.com/t/structuring-large-complex-django-projects-and-using-a-services-layer-in-django-projects/1487/22

If you want to know why service layers aren't such a good idea, read:

https://www.b-list.org/weblog/2020/mar/16/no-service/

Do use a global `tests/` for all apps

Do not put tests inside your app - tests observe their subjects from outside and aren't deployed to production. This is also discussed in A primer on Django project structure.

Big modules are usually preferable over packages

We can feel tempted to turn models or tasks into a package simply because there are many tasks or models. But having a big module isn't necessarily bad in itself.

Django will usually need to load your entire project, anyways. So there's no performance arguments.

The harmful part comes when we start to re-invent the whole domain/scoping problem inside foobar.models - if we already defined the scope of foobar, we now want to define a sub-scope? Is that even possible? Or will we simply end up with circular imports inside foobar.models?

Sometimes, it's easy or trivial to do. For instance, we could have:

foobar/
  tasks/
      emails.py
    sms.py

That looks good, right? ✅ In one module, we have tasks that send emails, and in the other one, the tasks send out SMS. If we need to do both, the caller will combine them.

But what if we would divide our models?

shop/
  models/
      messages.py
    purchases.py

You see, we would pretty easily end up in a situation where models.messages depend on models.purchases and vice-versa! ⚠️ Or we could have models that would fit in both places.

Navigate Locality of Behavior vs DRY

DRY is a great principle because it:

Gives us less code to test, review etc.
Makes it easier to avoid bugs when introducing new features

But it also gives us abstraction, which means that we end up designing functionality only for the sake of abstractions. You probably want to avoid that, unless you're writing a reusable library for external code bases.

And on the contrary, Locality of Behavior can result in more code and more to test and review + larger risk of bugs since there's more code to "remember" to change when refactoring. "Oh oops, I was updating the thing here, but forgot to update the same thing there".

But Locality of Behavior is great because it makes life easier when we make choices about where to put code inside a Django app.

Running Pre-commit in Docker

Sun, 05 Apr 2026 21:03:18 +0000

Make your "pre-commit" command run in a Docker container and reduce attack surface on your development system!

You probably know pre-commit which is a super-popular way to automatically install and run the required linters of a Git repository. It'll fetch all the required software and run it from an isolated environment, such as a virtual environment or a node environment.

Only problem: It's not really that isolated. So if any of these pre-commit hooks would suffer a vulnerability, they could have access to the secret files in the repository, your SSH keys or your various tokens for PyPi, GitHub, AWS, NPM etc.

As you might have noticed, supply-chain attacks are becoming more and more common. So the chance that a pre-commit hook falls victim seems evermore increasing. Especially given that exploiting a pre-commit hook is very attractive, given that it would gain the attacker access to developer's setups... the types of setups that usually also have access to lots of other exciting things.

I'm not trying to tip off attackers, I'm sure they already know this.

What can we do?

Obviously, if all the hooks could be packaged from trusted sources that would follow certain standards (such as Debian packages), we could start running linters in a more standardized way etc. But that seems a bit like wishful thinking right now. Pre-commit hasn't been trying to make it easy to install standardized packages, but more the opposite: To speed up the distribution of small, convenient automatic script-like features.

That does seem like something you'd want to run in a less trusted environment.

So the proposition here is to see how far we can get with Docker.

Try it out

I've written a little Docker image with some instructions here:

https://codeberg.org/benjaoming/pre-commit-docker-alias

What you get from that is basically a global alias for pre-commit that will work for all your projects. It contains a PyPi and NPM environment, so it's ready to handle these types of hooks.

More can be added, and we can figure out the basic day-to-day needs.

What are the catches?

I found only a small performance catch of like a 0.4s startup overhead on my laptop. This comes from spinning up the Docker container. It's nothing that you'll notice.

We should also be aware that the Git repository is mounted into the Docker container, which means you still have to secure any secrets that are exposed through the repository. I imagine we should stop storing secrets in .env files.

The other catches relate to current limitations to what you can do in the Docker image: It doesn't support Docker itself and some other language environments. But the images can be customized for this.

I encourage that you just fork my project and make it your own. It's not like the global image needs to do everything, we can manage different images for different projects and come up with a way to manage that,.

A primer on Django project structure

Sat, 18 Oct 2025 20:58:19 +0000

Here's an example recipe for a Django project layout:

src/
  apps/
    users/
  project/
    settings/
    templates/
    asgi.py, urls.py etc..
  manage.py
tests/
pyproject.yaml

That's the basic layout we're going to discuss now. But if you want to get started quickly, try running the prototype startproject.sh command that will bootstrap the above, with your project name as the parameter (source: Codeberg repo).

And food recipe blogs, notice how we STARTED with the recipe... make note that you CAN actually serve the recipe BEFORE the word soup 😎 🥗

The pitch

Whether your project is a quick'n'dirty one-off project or you're starting something with a longer trajectory, try to avoid stalling at universal decisions and instead use that time to design project-specific structures or simply start building. If your project is small or has low complexity, you can be happy with a simple structure like the above. If you expand to more complex needs, this is a good start that doesn't impose technical debt from the inception of a new project.

In this post, we'll discuss more principles for adding additional structure (UPDATE: and a blog post that's just about to be published here will dig even further into the architecture of Django projects and apps).

If you're new to Django and you don't know what exactly these various components in the structure are, there'll be some additional pointers in the below discussion of choices. Even if you're not new to Django, I encourage that you revisit these decisions and wonder "wouldn't it be nice to always bootstrap a project this way?".

At the time of writing this, DEP-14 and DEP-15 were accepted. These are both proposals to change things in Django: The former proposes a new django command for everything, and the latter introduces django new sub-command that will prompt the user about which starter template to use for a new Django project. So let's say the trajectory of this blog post is also to open the floor on how such a template could look and behave 🌠.

Here is what the startproject.sh offers:

Your project is a Python package from the beginning
You can bootstrap a Django project without installing Django firstly
It contains opinions on the Django project layout (detailed below)
uv, pre-commit and Ruff from the beginning

NB! (to be updated later) At the time of writing this, the owners of uv have sold themselves to OpenAI - this blog post is not an endorsement of future uv/Ruff

Ultimately (moonshot part 🚀🌕), we should let django new break new ground by combining its nice simple interface with a new tutorial that makes it easier for people to use and understand the combination of Django project and Python package.

🗯️ Comments, questions, thoughts? Please feel free to respond to the Fediverse announcement of this post.

Folders

This was intentionally made as a kind of baseline for a Django project. We will have a section for additional common problem and their solutions. But here are some explanations of the initial choices:

src/: We will put all our Python source code in a distinct folder, because your project WILL have other files and folders that aren't Python source code. You may or may not want to distribute a Wheel, but this is just a tidy and very simple rule. Polluting the root various packages is not what you want, trust me.
src/apps/: We have apps and projects in Django, those are very important distinctions that should trigger a Django developer to engage in long philosophical pondering. So in order to reflect this understanding in the folder layout, let's avoid the conflation of apps and projects (or actually "project" in singular).
src/project/: Almost the same thing, except see what I did here? I made the main project simply be called project, in singular. Yes, it could have been django_project or site or <organization_name>. But ultimately there is just one project and it's the Django project ☺ By simply calling it "project", we maintain the terminology of what makes a project be different from all your app domains, and no one needs to wonder "is this an app?" - no, it's a project!
src/project/settings/: You will of course need a settings module, otherwise your Django project cannot run. But from the inception of your project, make it a package so you can have a variety of settings for each of your fundamental environments (often that's development and production, maybe test and staging).
src/project/templates/: You should always anticipate at least on project-wide templates folder because there are templates that form the base of all your apps. Otherwise, you will end up storing project-related templates in your apps.
src/manage.py: This is the main entrypoint of it all, it could be in the root folder, but let's treat it as a part of the main src that should be installed on the target. This makes it possible to ultimately create a Python package and point to manage.py with an entrypoint. Do you wanna run acme runserver inside your VPS or AWS task or Docker Container? Sure thing, just configure the pyproject.yaml with these aliases.
tests/: Here is a very very important simple step: Always keep tests separate. We will get into how we can organize tests later. But do not mix them with the actual project, tests should be observing their subject (the application/project/software) from outside.
pyproject.yaml: Python in 2025 pretty much makes it possible to manifest every configurable aspect of your project in this toml file, so let's stick to that. This also means we can operate uv from the beginning to do things quickly, like managing your environment and adding more dependencies to your Python project.

Non-Django ingredients

It's possible that other tools such as environment and project managers like uv (or pip, hatch , etc - and combinations) and pre-commit (or the recent prek) will also fit very well when bootstrapping a project. Some people would even say that Docker Compose is a fixed ingredient in all their project recipies. Aside from that, there are commonly used Django dependencies like psycopg(3) and django-debug-toolbar. It means that we might have A LOT of considerations to present in a template or a tutorial. But these are more like variables, unlike the basic structuring of the Django project itself. Django itself can have more opinions about how to get people started, which is what the new django new command will offer.

This very adjacent to the idea of "batteries included": Get people started with developing their web project, including pytest, pre-commit, ruff and uv.

Distinction between apps and project

src/apps/ vs. src/project/

The following sentence is not very intuitive: We will stop thinking that a Django project consists of apps - the apps actually live outside of the project scope itself. That means we need to consider what a project really is, and what it means if the apps are not included? Well, a Django project is a central, single unit that combines and configures a set of apps. Those apps might be very targeted to work in the project, but ultimately, they are standalone Python packages - and even if their usage outside of project is unthinkable, we still maintain that position! The unintuitive part here will serve to help us define and design each app.

So you have these layers:

The Python project: Has everything, and is defined by pyproject.toml
The apps: Are Python packages inside the Python project, designed to behave as Django apps
The Django project: A Python package inside the Python project designed to be... a Django project!

We're not gonna mix these layers, so now we can stop worrying about Django apps somehow being inside a Django project. Because that is confusing! (Sorry to the people advocating the single-folder django app & project combination)

Remember also that outside of these layers, we also have "the IDE project", "the Git project", "the Docker Compose project" etc. So people are more or less familiar with the ambiguity of the word "project" - or let's just call it an inevitable condition.

App domains

Apps each have a domain: This is taken from good old domain-driven design. This should avoid the apps from overlapping, making it easy to decide what goes where, and avoiding circular dependencies. Apps may depend on each other one-directionally, but you should always try to make it so that dependencies aren't two-way (this might be impossible in some rare occasions). The user app is there from the inception as a good practice: You should probably always have your custom user model and avoid having anything in this app's domain intersect or depend on other apps. If you have a shop app, it will depend on the user app, but the user app should not depend on shop!

Discussing app domains is a positive problem. It helps you and your team understand your problem better. If your app domains are hard to define, it's very likely that the solution needs a bit more work - and hopefully if this is hard, it's also a fun challenge!

src/apps/api/

We all want APIs now and then, but do we create one API per app? Or a common API for all our custom apps?

Let's think for a moment: We have the Python project containing the Django project and a set of stand-alone Python packages. Do we have a simple API for each app? If that makes sense to you, you should definitely define APIs inside each Django app - maybe inside a sub-package like blog.api with nested views and serializers (if you're using Django Rest Framework).

But if this does not immediately make sense to you, my recommendation is to start differently:

Create a Django app simply called "api".

So you get:

src/
  apps/
    api/  # <- A unified app depending on all the other apps included in API views
    blog/
  project/

From this starting point, you have both the option to try to keep your APIs separate for each app domain or allow the intersection. But even better, you will have a mirrored solution in your tests where you can target all your API tests on this particular app. That can help to structure your project in general, not least since most APIs share a common codebase around authentication, rate limiting, caching etc.

Templates

If you're using your Django project as a fullstack project with templates, then template structures are very important. You don't want templates and templatetags to be inheriting across apps. The templating language has less support from IDE and linting, so it's important to try to stick to something that doesn't become complex.

The suggestion here is to ALWAYS have a templates directory for your project and to create a "base.html" here that each app will inherit from. Even if you're not targeting a distributed, reusable model, you should still organize these things in a neat, intuitive way. This makes it easy for other developers to guess your structure's logic.

src/
  apps/
  blog/
    templates/
      blog/
      base.html  # extends from "base.html"
  project/
  templates/
    base.html  # your whole site's base template

This is quite similar for static files.

Settings

As a baseline, we should always use environment variables, but there are a few things that aren't well-suited. But instead of unlimited settings permutations, we assume that there are a very few selected, eligible settings.

We ALWAYS want a separate development settings module. This will not only include DEBUG=True but also configure things like django-debug-toolbar, maybe a different database configuration, maybe eager triggering of Celery tasks - who knows. The point here is to AVOID following the 12-factor model strictly and end up with a jungle of environment variables. You may argue that it's also 12-factor to specify DJANGO_SETTINGS_MODULE in your deployment.

The main caution here is to avoid drifting. This can lead to developers to make wrong assumptions about the production environment because their development environment is too different.

We are intentionally NOT including the otherwise populare "settings/local.py" because this is often a source of developer bugs. A developer may have different settings from the rest of the team, and suddenly there are two sources that have to be debugged: The environment variables AND a local.py.

Celery/django tasks

Like views or models, tasks are a very specific kind of object in Django. It's a function that receives a set of serializable paramenters and perfoms its functionality from only that context. It happens outside of the request-response loop. It's important to scope tasks so they are easy to test and maintain. The conclusion here is to keep your tasks bound to each app's domain and have for instance:

src/
  apps/
    blog/
      tasks.py  # <- all tasks within the blog domain
  project/

Again, if you have dependencies between apps, make sure they only depend in one direction. A new shop app can depend on a users but not the other way. So you can have shop.tasks doing shop-related operations with users, but then you should not put shop-related things in users.tasks! Whenever app domains and their dependencies become tricky or complex, it hints towards defining new apps.

Tests

If you read this far: You might be inclined to skip tests in the beginning of bootstrapping your project, but you will need them soon. And the worst kind of test setups are the ones that don't distinguish between their various genres: e2e, integration and unit testing.

A good goal for your layout is to eventually create one test package per app. And you might as well also start out like that.

src/
  apps/
    blog/
  project/
tests/
  conftest.py  # <- Here we have global test fixtures
  blog/
    conftest.py
    e2e_tests.py  # <- test your views here and let everything execute. Few but expensive.
    unit_tests.py  # <- test individual functions

This is really just an example: You might also want to create view_tests, model_tests etc. But still make sure that you can grasp the different genres of tests:

e2e: Everything runs, your database layer, views, Celery tasks, external (mocked response) calls etc.
integration tests: Rarely used in Django projects, but you might fancy this to allow certain layers to be called with internal functionality executed. This is especially useful if you decide that your unit tests should not access the database.
unit tests: A simple function is tested, external calls are mocked

The idea here is still to ensure that you design every test case with some understanding of its scope: You should not test a simple model method by calling a view, and you should not write loads of assertions on the internals of a model method if you're actually testing the view.

This all boils down to finding the value and motivation for writing tests. Try to find yourself doing test-driven development as soon as possible, but if it doesn't make sense, don't do it... yet... but still, at least squeeze in some e2e tests from the beginning, they usually provide a payoff quite quickly

Enough about testing, it's a whole category of its own 🤪

Q&A

Should I start with a `users` app and `AUTH_USER_MODEL`?

Django provides a simple User model from django.contrib.auth. However, once you have started using it, migrating to your own custom model becomes tricky and most people resort to having a relation to some kind of UserProfile model.

That's why I recommend to start all projects with your own users app and define settings.AUTH_USER_MODEL to point to that model. You can read more about this in the Django docs: https://docs.djangoproject.com/en/stable/topics/auth/customizing/#specifying-custom-user-model

In order to reduce how opinionated this project layout is, I kept it out, and I think these sorts of choices are more appropriate for a tool like cookiecutter.

Should I create packages for views and models?

I've seen many examples where a Django application have for instance a whole package app.models full of nested models, each with their own domain. Having domains inside the app domain is easily a sign that you need a new separate app, but it can also be a sensible and intuitive way to split up models and views. Maybe you have models.abstract or views.api, views.public etc.

I'd prefer my models.py to be lengthy, rather than having to wrestle around with __init__.py imports and from .models.submodule import ModelName.

Service layers?

It's outside of the scope here to discuss the idea of service layers. I might discourage it, and I certainly don't feel like something like an apps/blog/service.py would be appropriate in a template.

Shared libraries and abstract models

It makes sense to have a common or shared app, but seeing that it's not likely to have any concrete models, it's best to place it outside of your apps (so it's not a Django app) and project (since it's not really part of the Django project). That could look like this:

src/
  apps/
  project/
  shared/

Monorepos?

Not far from this discussion maybe: it's true that backend API projects could discuss whether they live alongside the frontend etc. I

Why "project" and not "config" or "site"?

There are some drawbacks to using these other names:

config or conf: I think that config/templates is wrong, if we allow an HTML file to be considered "configuration", then anything is configuration or we have to invent a NEW location for project-wide templates, static data, backends etc. It's not a terrible convention, definitely better than nothing. It's used by the project django-new for bootstrapping Django projects and apps.
site could also work but I think project works much better because we already have a concept of a "Django project", whereas a "Django site" would be something completely new and without clear advantages.

Why is `tests/` separate from `src/`?

In this article, we knowingly recommend to maintain all test-related code in its own tests/ folder/Python package.

Maybe you've seen layouts where Django applications each have their own app/tests/ folder. This is a bad idea because your tests will be built into your application. If you have test fixtures, they will be loaded into your deployed application image. If you do a mistake, the entire test stack might be inspected/imported.

A common "lazy" mistake of maintaining test code together with the application, is to start mixing test requirements with application requirements, too. This could ultimately result in pytest or playright ends up in production images.

Overriding Django middleware, db routers etc

I would propose that anything you extend and override from inside Django is placed in your src/project/ folder. We can see it as a kind of configuration. In certain cases (such as authentication) you might have an app's domain fit perfectly, but in most cases, you need a one-off customization that fits well in src/project/.

Links

Using template partials to avoid lots of files:
https://indieweb.social/@adamghill/115329417459835209

Against service layers in Django
https://www.b-list.org/weblog/2020/mar/16/no-service/

Why do we need apps?
https://forum.djangoproject.com/t/why-do-we-need-apps/827/28

HackSoft's Django Style Guide
https://github.com/HackSoftware/Django-Styleguide

Deploying Django on a VPS

Containerizing applications comes with its own set of issues, but in a world where application and cloud platforms are pretty much the norm, it feels like we have to argue the case of using a VPS (virtual private server) in terms of its advantages. The tables have turned, so rather than argue why to containerize our applications, it seems like we need a little reminder about the advantages of a VPS:

Share monitoring of several small projects and services
Keep important services updated using a central package manager
No Docker debugging
Webhook automation for fast, downtime-free deployments without rebuilding containers
SSH access "live in production"
Easy cron jobs

How-to and ingredient list

As an example, here's a tiny how-to where I'll go through some steps. What we need is:

(optional) if you are firm with Bash, write your own idempotent bootstrapping script
An Ubuntu or Debian-based VPS with systemd
Git + other essential dependencies
Nginx and certbot
Postgres (assuming you want a database)
Gunicorn
Webhook for continuous deployment
Other nice things

1. Optional: Write everything in a script

I highly recommend that you develop your own private script for bootstrapping a Django project on your VPS. This will help you to document what you do, and to reproduce it in the future.

Also: Write the script to be "idempotent". This means that the script can always run again and will to some degree simply update an existing setup. The main advantage is that you can continue to develop the script while using it to bootstrap a new project.

If it's a Bash script (recommended), then make sure it has set -e so it will crash when a step fails.

2. A Debian-based VPS with systemd

In this blog post, it's assumed that you have something with Debian on. Here are some things you should do to harden the VPS in general - even if it was to run Docker, you should still do this. There's basically now exception:

Use unattended security upgrades: sudo dpkg-reconfigure unattended-upgrades
Set your timezone: sudo dpkg-reconfigure tzdata
Run SSH on a non-default port
Add your own user to the sudo group and disable the root user
Don't allow password-based logins via SSH (use SSH keys)
Run locale-gen en_US.UTF-8 - often this is missing. If you have other locales, make sure to generate them, too.
Remove snapd (if you use an Ubuntu image)
Consider fail2ban

3. Git + other essential dependencies

When bootstrapping the new VPS, there are some pretty common dependencies that you will need later. I'll just leave this single apt install command, but I highly encourage that you document your own needs and write them in a script:

apt install git python3-pip nginx certbot python3-certbot-nginx gettext postgresql postgresql-server-dev-all

If you want to use pip instead of uv, I recommend using the system package python3-virtualenvwrapper so you can issue commands like mkvirtualenv directly in your VPS. This how-to assumes you want to use uv.

This list has gotten shorter over time. Back in the days, Pillow (or Python-PIL) had a bunch of system requirements in order to compile things.

4. Nginx and certbot

Now we need to setup Nginx! We're gonna set it up to use SSL and you need to update DNS settings to point to your VPS before getting started.

Nginx has a brilliant structure for VPS hosting many sites side-by-side:

/etc/nginx/sites-available contains ALL your configurations
/etc/nginx/sites-enabled contains symbolic links to the enabled configuration files from ../sites-available

We use certbot and LetsEncrypt to fetch a certificate. I recommend setting this up in a way where you have a default server to listen on port 80 and thus bootstrap your first SSL certificate for a server on port 443.

Do a quick mkdir /var/www/certbot, then edit /etc/nginx/sites-available/default to say the following:

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    # Allow certbot ACME root
    location /.well-known/acme-challenge {
        root /var/www/certbot;
    }

    # Redirect all traffic to https://
    location / {
       return 301 https://$host$request_uri;
    }

}

Now run systemctl reload nginx and let's issue our first certificate, supposing the DNS setup is correct:

certbot certonly --cert-name mysite.com -d mysite.com  --webroot-path /var/www/certbot

Now let's add some general SSL settings in /etc/nginx/includes/ssl.conf:

# Listen on 443
listen 443 ssl;

# https://wiki.mozilla.org/Security/Server_Side_TLS
# http://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/

# Consider even removing TLSv1.2
ssl_protocols TLSv1.2 TLSv1.3;

# We will prefer a compatible client-chosen certificate
ssl_prefer_server_ciphers off;

# Recommended
ssl_ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1:!AESCCM;

# Enable the below on Nginx 1.19.4 and later, disable if Nginx fails to start
# ssl_conf_command Ciphersuites TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256;

If this goes well, you're now ready to add the fist SSL configuration and then we can expand that later with our Django application. But for now, you can add the following in /etc/nginx/sites-available/mysite.com:

# Redirect traffic on www.mysite.com to mysite.com, still host acme challenge
server {
    server_name www.mysite.com;
    ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem;
    include "includes/ssl.conf";

    location /.well-known/acme-challenge {
        root /var/www/certbot;
    }

    location / {
        return 301 https://mysite.com$request_uri;
    }
}

# Here is your actual mysite.com configuration
server {
    server_name mysite.com;

    include "includes/ssl.conf";

    # Check if this certificate is really served for this server_name
    # http:/serverfault.com/questions/578648/properly-setting-up-a-default-nginx-server-for-https
    if ($host !~* "(www.)?mysite.com") {
        return 400 "this is an invalid request";
    }

    ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem;

    # Adjust the allowed upload size
    client_max_body_size 50m;

    # location /favicon.ico {
    #     alias /var/sites/mysite/git/staticfiles/img/favicon.ico;
    # }

    # location /robots.txt {
    #     alias /var/sites/mysite/git/staticfiles/robots.txt;
    # }

    # location /media  {
    #     alias /var/sites/mysite/git/media;
    # }

    # location /static {
    #     alias /var/sites/mysite/git/staticfiles;
    # }

    # Webhook reverse proxy
    # location /hooks/ {
    #     proxy_pass http:/127.0.0.1:9000/hooks/;
    # }

    location / {
        try_files $uri @proxy;
    }

    location @proxy {
        # TODO: Add gunicorn reverse proxy
    }

}

Cron job for certbot

What's nice about a VPS is that you have a cron daemon builtin. So let's add a rule that automatically renews our certificate.

Try running certbot renew --post-hook 'service nginx reload' to ensure that it works. Adjust as needed.
As root, run crontab -e
Make sure the following line appears (example renews at 1 AM everyday and adds --quiet to the command): 0 1 * * * certbot renew --quiet --post-hook 'service nginx reload'

5. Postgres

If you need Postgres, here are some nifty commands to run either manually or add to your script:

# Assuming you fetch the username from command line arg $1
db_username=$1
# Assuming dbname is the same
postgresdb=$db_username

su postgres -c "createuser -P $db_username"
su postgres -c "createdb --owner $db_username $postgresdb"

6. gunicorn

Some comparisons between gunicorn and uWSGI on a VPS

As you might have heard, uWSGI is in maintenance mode. It means you should consider to use a different option for future projects. As Gunicorn through uvicorn also provides a nice path towards support of ASGI, you can consider this a good bet for the future.

However, because uWSGI has such a lot of history, it also has a lot of things we should pay attention to, especially on a VPS:

uWSGI centralizes configuration which makes things easier to manage and reuse. We should manage our gunicorn configurations in a way that's easy and so we don't lose track.
uWSGI's debian package and Nginx plugin comes with bundled in Nginx parameters in /etc/nginx/uwsgi_params - we should also try to centralize our gunicorn parameters.
Some reverse proxy headers are better defined out-of-the-box with uWSGI. We need to pay attention, especially when migrating existing projects.
uWSGI is part of the operating system and integrates with systemd - we need to create and enable our own systemd scripts for each of our gunicorn services.
Your application does not provide the application manager? Does that make better sense?

Adding a Django middleware

Because gunicorn is a reverse proxy, having Nginx in front isn't trivial. Many things are addressed in the Nginx settings and work well, but you have to pay attention to especially the fact that your HTTP request meta data have passed through one layer before it's handed to your web application. I recommend that you add an additional middleware in Django to handle the missing request.META["REMOTE_ADDRESS"] (and any other rules you might add), which is expected to contain the client's IP:

def reverse_proxy(get_response):
    def process_request(request):
        if "x-forwarded-for" in request.headers:
            request.META["REMOTE_ADDR"] = request.headers["x-forwarded-for"]
        return get_response(request)

    return process_request

7. Webhook for continuous deployment

8. Other nice things

Forward system mail to your actual mail account! Setup your email daemon to receive notifications, for instance when cron jobs fail.
fail2ban
Log rotation
goaccess

A Venn diagram of Open Source governance

Tue, 02 Jul 2024 15:05:03 +0000

Duplicate form submissions and how to handle them in Django

Sun, 28 Jan 2024 14:10:58 +0000

Let's have a look at duplicate form submissions, understand the problem and fix it with a nice server-side solution. It's a common problem that almost always needs to be addressed - whether you are building HTML-over-the wire or a backend/frontend architecture, you will have the same issues:

User clicks the same button twice.
User navigates back and re-submits a form that we already processed.
User reloads the response page and resubmits same POST/GET form data.

Typically, this is because of some disturbance in the network or your own server having hiccups that prompt the user to retry. The duplicate form submission can then trigger:

Duplicate database objects
Duplicate user notifications
Duplicate anything_really
False-positive UI validation errors because of uniqueness constraints (first submission was valid!)
Further unnecessary server-side errors in case of unhandled uniqueness errors

Likely, the worst part of this problem is already fixed with the "POST-response-redirect" pattern, where the post target responds with an HTTP 302 and sends the user to a "thank you" page. Since it's built into Django's generic form handling views, you've probably already done that part.

Desired properties

You can skip this if you don't care about what the solution aims to provide, but I'd love to hear back about anything that's been missed here!

MUST process duplicate form submissions exactly once.
MUST allow user to perform duplicate form submission and succeed journey.
MUST process non-duplicate form submissions (defining what "duplicate" means is the solution to this requirement).
MUST allow duplicate form data from different user sessions (same as above, but a good illustration).
MUST function across multiple application workers / containers / threads.
SHOULD give user a meaningful response when submitting duplicate form data.
SHOULD be generic and easy to reuse.

In terms of multiple application workers and containers, there are race-conditions in which I actually haven't solved the above 100%. UPDATE: See final discussion.

Client-side disabling the submit button onsubmit

Here is a really simple snippet that resembles a common solution. You can add to avoid users accidentally or deliberately clicking the submit button multiple times:

// When a form is submitted, disable all its submit buttons
$(document).ready(function() {
    $('form').submit( function(event) {
        $(this).find(':submit').attr('disabled', true);
    });
}

But STOP: What could possibly go wrong?

The user never gets a second chance to submit the form, in case they had connection issues. So we'd need a timeout or some error handling to restore it to enabled state.
If the user does submit but receives an error in the browser (network error, Wi-Fi login etc) and they press "Back" in the browser, the button is still disabled!
If you use name and value properties to know what button the user clicked, this will be disabled before submitting the form.

Because of all these issues, my opinion is that it's unfeasible to solve this with JavaScript, so let's just make the server understand if the form was submitted twice and give a meaningful response. It's a bit the same as when over-investing efforts in client-side validation - sure, it's nice (if implemented correctly), but you need to do it on the server anyways.

Database uniqueness constraints

Another preventive measure can be to add uniqueness constraints to your database. If you can do this, you should. And then you should handle these errors gracefully.

Here's some code that we might use to handle duplicate form submissions:

class MyCreate(CreateView):

     def form_valid(self, form)
         # Try to save the form, assume that IntegrityErrors from the database
         # are because we already saved the same form. 
         try:
             obj = super().form_valid(form)
         except IntegrityError:
             messages.info("This form was probably already saved.")
         return redirect("thank-you")

Notice though, that we don't actually know if it was the case that the form failed because it was a double-submission? We might also have form validation that kicks in before we try saving the object. In any case, handling the constraint error at this level, we risk informing the user that their form was saved - when it actually wasn't! We need to know that it was saved.

We can also do this while validating the form. In this case, we block the form and the user receives a validation error in the UI. But then the validation error might shadow the initial successful submission of the form! Remember that we are dealing with the problem of double-submissions.

So what can you do?

Let's get started with an approach for rejecting form POST data that we've already processed (it's not quite perfect though):

class MyCreateView(CreateView):

    def form_valid(self, form):

        # Unique key to store session for this view
        session_form_hash = f"form-submission+{self.__class__.__name__}"
        # Unique key for storing the pk when the object is created
        self.session_last_saved_pk = session_form_hash + "-pk"

        # Calculate hash of the POST data
        excluded = {
            "csrfmiddlewaretoken",
        }
        this_post_hash = hash(
            tuple(
                sorted(
                    (k, v) for k, v in self.request.POST.items() if k not in excluded
                )
            )
        )

        # Previously calculated hash
        previous_post_hash = self.request.session.get(session_form_hash)

        # Form has already been processed!
        if this_post_hash == previous_post_hash:
            # Fetch the previous object and return a helpful message to the user
            self.object = get_object_or_404(MyModel, pk=self.request.session[self.session_last_saved_pk])
            messages.warning(
                self.request, "This form was submitted several times. It has been processed just once."
            )
            return HttpResponseRedirect(self.get_success_url())
        else:
            response = super().form_valid(form)
            self.request.session[session_form_hash] = this_post_hash
            # self.object is defined when using ModelFormMixin
            self.request.session[self.session_last_saved_pk] = self.object.pk
            return response

We're almost there:

Before the form is saved, we start by hashing its POST data (excluding the csrfmiddlewaretoken which can change). We're sorting the keys before hashing, so we can say that the hashing method is consistent.

We store the hash with a session key that's unique for this particular view. This means that we can detect duplicates that are unique for the current user's session and we can also share it across application workers/threads/containers.

Finally, we check if any existing form POST data for this particular form and user was hashed. If not, we proceed to saving the form and marking it saved. If it was already saved, we inform the user that the form was saved on the thank-you page and add a message that we registered duplicate submissions.

But we only do this in the form_valid view. If that's good enough, you can stop here. But if you have constraints and validation rules that would make the second form submission invalid, it will trigger the final form submission to come back as invalid. The user won't know that the first form was correctly saved.

Icing on the cake 🍰️

We're gonna continue to hash and save our new object's PK in the form_valid() method, but we are going to move the duplicate check to the post() method. That way, we can avoid issues with invalid form data when it's due to uniqueness constraints kicking in. In fact, we just skip processing and validating the form all-together when we've already seen it.

Win-win! 💯️

class MyCreateView(CreateView):

    def post(self, request, *args, **kwargs):

        # Unique key to store session for this view
        self.session_form_hash = f"form-submission+{self.__class__.__name__}"
        # Unique key for storing the pk when the object is created
        self.session_last_saved_pk = session_form_hash + "-pk"

        # Calculate hash of the POST data
        excluded = {
            "csrfmiddlewaretoken",
        }
        self.post_hash = hash(
            tuple(
                sorted(
                    (k, v) for k, v in self.request.POST.items() if k not in excluded
                )
            )
        )

        # Previously calculated hash
        previous_post_hash = self.request.session.get(self.session_form_hash)

        # Form has already been processed!
        if self.post_hash == previous_post_hash:
            # Fetch the previous object and return a helpful message to the user
            self.object = get_object_or_404(
                MyModel,
                pk=self.request.session[self.session_last_saved_pk]
            )
            messages.warning(
                self.request, "This form was submitted several times. It hans been processed just once."
            )
            return HttpResponseRedirect(self.get_success_url())

        return super().post(request, *args, **kwargs)

    def form_valid(self, form):
        response = super().form_valid(form)
        self.request.session[session_form_hash] = this_post_hash
        # self.object is defined when using ModelFormMixin
        self.request.session[self.session_last_saved_pk] = self.object.pk
        return response

By storing the POST hash after the fact of validating and saving the form, we are guaranteed against wrongly blocking a form submission. And we'll know that it was previously validated and saved already before validating the next form submission, catching any duplicate form submissions.

What's next?

Because I think that any project would need to adjust this to their own definition of what a "double submission" is, I'm going to leave this here for now. But we might be able to brew together a nice view decorator.

UPDATE: There are a couple of edge-cases with respect to high load and concurrency that you might consider:

Race-conditions: Processing the form in the database can take time, leaving a gap for double submissions to enter while another submission is being saved in the database by another worker/thread/container.
Session and cache invalidation: We use a session to store form hashes. If you have a LOT of sessions, consider that the timeout for a session is much higher than what is needed to guard against duplicate form submissions. So you might use a cache item directly and base it off the current session ID, but setting a timeout to some amount of minutes.
In any case, remember to always run clearsessions.

I didn't have any large-scale or high load/concurrency use cases on my radar, but if you have any inputs, join the conversation in the Fediverse.

Code globally, meet locally

Tue, 19 Dec 2023 16:24:06 +0000

I recently shared initial thoughts around Local conferences: Big potential! In this follow-up blog post, I want to share practical information on the specific format that we've been organizing Django Day Copenhagen around -- aaaand finish with some good old motivation 🥕️

With a practical focus, I hope that the blog post can make people consider for real that doing a small local conference is easy, fun and worth-while!

In October, we hosted a little event called Django Day Copenhagen. It was the 4th time that we hosted the event, which attracts around 50-60 people to the venue and an additional ~100 live streamers. After the event, the live stream is watched many times more (2022's live stream had 1.3k views), and each of the talk recordings already have some 100-500 views.

More importantly for this blog post, we can compare with our experience from a large international event since we hosted the DjangoCon Europe 2019 in Copenhagen, a 3-day event for 400 people. So we know a lot about larger events.

Having local conferences isn't meant as an alternative to all larger conferences. It's meant as a reminder to communities that unnecessarily converge on only large conferences: There are alternatives for community and educational events that we are missing out on. I'll get back to this at the end of the blog post.

What is the concept?

Let's start with a definition of the format of our small local conference:

A 1-day single-track event, for which you probably need 6-10 talks to fill the day.
A smaller venue with room for 50-150 people.
We want to ultimately attract a local audience.
...but our speakers might travel from abroad, and so will people with an admirable motivation.
We keep down costs, for instance we consider local socio-economic enterprises for catering and municipal/cultural spaces as venues.
We use online pre-sales for tickets.
Corporate supporter tickets (=no sponsorship program!)
No conference booths.
Breakfast and lunch included at the venue.
Volunteer-driven (except recording and streaming).
Online streaming
Small budget for speaker transport or accommodation.
Tiny budget and margins for error, meaning that we don't run big financial risks.

...and with that said, I think everyone should adjust this to their needs and their venue.

Some specific goals

Along with the conference format, it's always good to keep in mind why you're interested in doing a conference, but more importantly a local one:

It's fun, you meet a lot of people, and you can stress less with a local event.
Reduce the climate impact of conferences: Most emissions caused by conferences are from attendee travel. Large conferences are unsustainable for our planet.
Improve economic inclusion: You might already be affected by the global economic depression, but many people were already excluded from conference events before recent events have caused a decline in conference budgets.
Strengthen socializing and local organizing: The Open Source movement as a whole will benefit from more local initiatives, as in the early days of "user groups" and later days of Meetups.
And all the usual goals of a conference: Strengthen the community and interests in your favorite technology, for the benefit of a new generation, renewed commercial interest, innovation etc.

Steps to get started

Let's go through some practical ideas about how organizing a small local conference could look. This doesn't mean copy this "1:1", there's guaranteed to be parts that you can do better or differently or not at all.

Financial body

Let's start with the most simple aspect of the setup: In order to sell tickets and pay for things, you should probably consider running this through a real legal organization (as opposed to a private person). Typically that means a member-based association or non-profit.

Note: You might be able to skip all this, if you can identify an existing financial body to run the event! Maybe there's an existing Open Source organization in your area that you can ask to host this?

If you are just getting started, then set aside time to register the organization and its bank account. But don't let it stop you, just ensure you have enough time for the bureaucratic procedures that are necessary before you can sell your first ticket.

By working as an association, we have become eligible for cultural and municipal venues, which are exclusively for non-profit activities. This is a big advantage. At the same time, an association can act as a company and sign up for various banking and online services that we need.

If you want to do this as a private person, it's possible but not very recommendable. It will severely limit your reach. It will be hard to sell tickets online, and many attendees prefer to have an invoice. You'll have to be creative on a level that's going to take away focus, and it's probably more suitable to consider doing a "meetup" format where all costs are handled through donations or contributions from local companies.

Corporate Supporters: Skip the sponsorship program

Having a real sponsorship program implies a lot of tasks. You need to define exactly what the sponsor gets in terms of visibility and implement special ways to care for this.

We simplify this work by offering a simple one-size-fits-all "Corporate Supporter" ticket. The ticket gives a standard experience for all supporters:

3 sizes: 1 ticket, 3 tickets and 5 tickets
Logo on website
Gratitude during the event
Logo on sticker sheet (for the largest level)
Access to the chat to post job offers

Party

We finish Django Day Copenhagen with a party. Everyone gets some time after the event to go and get dinner somewhere and recharge, then we meet again after a 2 hour break for drinks and pub quiz.

We have done this in 2 flavors: Either we have a venue that can also accommodate a party where we sell our own drinks in the bar, or we book a big pub somewhere close to the venue.

It's a big advantage to host the party: It's fun to be in the the bar for volunteers, and the profits from hosting a party are enough to ensure that the event finishes off with a little extra financial boost.

The disadvantage of hosting the party is that you'll have to shop everything for the bar. For all our events, we shopped all drinks and everything that wasn't sold at the party has been sold off afterwards. So we never had any issues with unsold drinks. To generalize this: Running a support-party is fun and very risk-free.

Thoughts on improving at Django Day Copenhagen

Outreach: We didn't do any direct outreach for the event, despite our knowledge of several companies using Django that do not attend.
Posters were too much effort: We printed posters but people weren't interested. We conclude that probably it's not worth the money and effort to add physical merchandise on this level.
We also put up posters at 2 local universities but no one attended the event because of this. Advertising the event at universities requires a presence there, for instance by visiting relevant lectures and inviting students. We consider making it extremely cheap for students to attend, since the generational overlap is critical.
The evening before the event is for watching TV: As an organizer, I fell into the trap of doing stuff the evening before the event. I simply don't have the extra deposits of energy currently to pull through a full-day event like this without getting an insane headache. So I would say that anything that isn't done before 19:00 on the evening before will not get done.
The ticket sales should close no more than 3 days before the event, as many people seem to forget to buy a ticket. We introduced "late bird" tickets which tackles the issue of people buying too late by offering them an option without food. The ticket doesn't include any discount, so there's no reward for buying too late.

First steps?

To do all this, you need to assemble a team, and you need to organize all tasks on a practical level. Here are some example columns for a project Kanban board:

Pre-launch tasks: Organization and venue
Launch: announcements, ticket sales
Before event
Outreach
Content and talks
Venue
Day before
During the day
After the event

Daniele Procida has previously written a very nice Community conference organisers’s handbook, however this guide is written for large events and will probably make organizing a conference look overwhelming to most people looking to organize a small 1-day event.

I would like to offer to do a simple practical handbook for getting started with Local conferences. If you're interested in seeing a more practical step-by-step handbook, please get in touch and motivate me!

Vision: Less large international conferences? ✈️ 🏨️ ➡️ 👣️ 🛖️

So, to finish off this long list of practical stuff, let's just talk about the need for large conferences vs. smaller local events.

Let's start by thinking about our technology as if it was a cultural event. Think about all these big open source projects like Django, Ruby on Rails or Laravel. Or if you like, their language ancestors Python, Rust, Go, JavaScript etc. Or even the bigger topics like Security, DevOps - or the more common "Web". Conferences happen on big stages so important speakers can reach large audiences with their important subjects. Afterwards, there's maybe some time for questions and the speakers can mingle a bit.

The audience and the speakers all travel to one location somewhere on the globe or on their continent.

Would this work in a cultural space? No, not really. Artists travel to their audiences, although as mentioned in my previoues blog post, a recent episode of BBC's The Climate Question concluded that even though it's the case that artists tour around, then the most significant climate impact of music festivals remains: audience travel. So that's what we need to focus on.

Adding calls for more diversity, more first-time speakers etc., I propose a hypothesis like this:

If our tech communities grow more local communities, more speakers will appear. If we have more speakers, we can afford a conference model with more speakers available to travel to local conferences, and audiences can converge on local events rather than big international events.

This is how cultural events work: Musicians travel, stand-up comedians travel, circuses travel etc. The show comes to your local community, and so should tech events.

By growing a local tech scene, we can adapt the whole system to depend less on expensive and climate-intensive shit like airplanes and hotels, and we can build more sustainable communities, both in social terms and in terms of the environment and the future economy of tech.

A settings pattern for reusable Django apps

Sun, 10 Dec 2023 19:40:06 +0000

Local conferences: Big potential!

Thu, 05 Oct 2023 19:46:48 +0000

Incoming blog posts!

Tue, 26 Sep 2023 09:47:49 +0000

Why I worked in the Danish public sector and stopped

How to actually install Node-js on Debian/Ubuntu

This post is:

A rant and
The simple instructions

The tl;dr rant

In order to install Node.js, you should execute a 330 line shell script directly from their servers. It's done this way such that the end user doesn't have to understand what's going on, and such that strange edge cases from old systems can be automatically fixed.

This is part of a larger trend of piping unread/unverified contents from the HTTP to a sudo-privileged shell. The hypothesis seems to be that users of all levels don't need to understand what is actually going on.

In order to for instance build an efficient Docker image that contains Node.js, you have to reverse engineer the 330 lines of code to understand the essential parts. Which is actually just 2 things (getting to that after the rant).

The security concerns:

Responses on forums and Stackoverflow channels, where an attacker can manipulate official instructions from curl -sL https://install.hype.io | sudo -E bash - to curl -sL https://install.malicious.io | sudo -E bash -.
Domain renewal/hijacking? Nodejs is a large organization that probably will be sure to renew nodesource.com, but what about the overloaded BDFL maintainer that wants to be with the cool kids, loses interest and has instructions all over the internet with curl -Sl install.bdfl-project.io | sudo -E bash - ? Next thing we know, having a Wordpress on a website that used to contain documentation (for instance, pipenv.org got squatted) will be that the "official" installation script is replaced by malware.

The scaled up inefficiency:

Moreover, they can't seem to decide if you have wget or curl in your system.. so it automatically finds out for you. Facepalm.

On a clean Ubuntu/Debian system, all that will happen is this:

# Trust their PGP signing key
wget -qO- https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
# Add the binary release repository
echo 'deb https://deb.nodesource.com/${NODEREPO} ${DISTRO} main' > /etc/apt/sources.list.d/nodesource.list
# For some reason, add the source repo, too, even though 99% of people won't use that
echo 'deb-src https://deb.nodesource.com/${NODEREPO} ${DISTRO} main' >> /etc/apt/sources.list.d/nodesource.list

Show me the code

Node.js on Ubuntu, adjust as you like.

# Trust their PGP signing key
sudo su -c "wget -qO- https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -"
# Add the latest 13.x binary release repository for Ubuntu bionic (18.04)
sudo su -c "echo 'deb https://deb.nodesource.com/node_13.x bionic main' > /etc/apt/sources.list.d/nodesource.list"
# Update apt
sudo apt update
# Install nodejs
sudo apt install nodejs

Taking back Firefox from the Internet

Firefox wants to connect to quite a lot of services. And it's highly recommended to read the article How to stop Firefox from making automatic connections, in which Mozilla explains and discloses a lot of these.

The article gives an impression of how contemporary software developers perceive their authority or privilege to access the user's Internet connection and whatever environmental footprint follows from these pingbacks, updates etc.

I'd like to point out the following:

These decisions take the user's bandwidth for granted by default
Connection retries, polling intervals etc. can be arbitrary and non-configurable.

Taking a user's bandwidth for granted

Recall that Firefox isn't the only software installed on a device or computer. Nowadays, software developers increasingly see their projects as outside or above an ecosystem and want them distributed in entire sandboxes (Flatpak, Snap etc). The decision of Firefox surely isn't unique.

And one of the peculiar things about this decision is that most systems already have for instance the network hotspot detection built-in. I have seen it on all major platforms: Mac, Windows and Ubuntu. Firefox still floods "detectportal.firefox.com" with requests.

Why I work for Learning Equality

Sat, 17 Feb 2018 00:14:46 +0000

Opinions and motivations of a tech-skeptical software developer @ an NGO working in education

Building a future with equal access to learning opportunities (via technology!) is a long and winding road: We need to consider sustainable scenarios for our technology and face up to true challenges — not be tempted by hype or simple fixes.

Anecdotes from when things were easier

About a decade ago, when bandwidth was a sparsity in rich and poor countries alike, designing and developing offline-friendly technologies was prioritized. Software ran on computers without internet access, things got mirrored on local networks etc. Not least, we were used to sharing documents and media with CDRs and USB flash, a.k.a. The Sneakernet.

Almost any computer or laptop could be opened with the same screw driver.

From my experience of working with schools in Malawi since 2011, offline education technology support has been an increasing challenge. Open access contents, educational resources and information databases have become more and more dependent on online platforms. One example is Khan Academy. It’s a highly complex and interactive website (with a simple UI!), only available to people with internet access. But back in 2011, it was a static website that could easily be downloaded and made portable on a hard drive and shared on a local network. Khan Academy provides great content, and they allow other actors such as Learning Equality to extend this content into the offline domain through a permissive license. That’s how KA Lite was started (and eventually how I came to know about Learning Equality and join the team to help build platforms for sharing educational resources offline).

Technicians (in Mzuzu, Malawi) installing software, replacing faulty hardware, instructing teachers: they are truly enablers, because they create the necessary environment for all the changes that advocates of a digital education would like to see. We can talk about technological enablers, but it’s actually people who maintain it, explain it, and configure it, who are key. (Photo: Benjamin Bach, CC BY 2.0)

Back in 2011, I was working with FAIR Denmark, and we left 4 refurbished rack servers running real Wikipedia (not just a static copy) on Ubuntu+Apache+Mediawiki+MySQL. They were installed in secondary school centres, serving approximately 50 computers through a cabled Ethernet. It had taken months to download and make Wikipedia work, with its 600+ extensions and ridiculous amount of image files. Finally, people could see photos in their original resolution and complete a free-text search in less than 10 seconds. It was actually pretty cool, because it was the same experience as having a real Internet connection: The slow search was compensated by the fast local bandwidth. We thought it was fine, but lo we forgot to calculate how fast Wikipedia was growing. 2 years later, it was evident that images could no longer fit on a hard drive, and the database was too big for text searches on a simple desktop server.

In 2016, I visited a centre that had originally had its server installed in 2011. Mponela and Malawi in general are dusty and warm places, and the hard drive with the database and all the images was an external USB 2.0 drive, consumer-grade. Surprisingly after 5 years, the rack server still worked, and the Wikipedia contents from 2011 were still showing (full disclosure: after 6 years, it went to lunch). Classroom computers that hadn’t developed hardware faults were still booting Edubuntu 10.04. Other places had maintenance and hardware swaps, but this particular server was the real deal. Stable software and stable hardware ❤.

The consideration here is not the encounter of an abnormally well-functional set of hardware and software, but real evidence of something that’s possible: Imagine what can be done educationally when an ICT centre Just Works™! Imagine a software and hardware platform that just spews out educational contents, and does so for years and years without any need for software updates, adapting to new functions or user interfaces. Just uninterrupted and simple delivery of educational contents.

2011: Refurbished rack servers were available in abundance, and there was a low complexity of distributing online through local networks. So we did that, but online resources have gotten more complicated since then. Also: Almost any laptop and computer could be opened only with one type of screw driver. (Photo: Benjamin Bach, CC BY 2.0)

Lots has changed since then: It’s not possible to scrape huge websites like Wikipedia without having a customized platform. Ubuntu no longer fits on a 700 MB CDR. Application memory footprints are increasing (you need at the very least 1 GB to run a modern desktop), and user interfaces are changing faster than educational books are published. Developers are shifting focus from the desktop applications over to web platforms, so for instance we no longer have Edubuntu, the education-flavored version of Ubuntu.

Most experiences and software setups deployed some 6 years ago are now obsolete. But I have to say: Back then, things were more stable, changed less frequently, and were less complex in their nature.

The world back then kind of didn’t need Learning Equality, but now it certainly does!

Complexity on the rise

I’m skeptic about most things that relate to technology, but working for Learning Equality is not a compromise. I’ve seen many dusty computers providing little or zero value for recipients in development projects, and I have also seen digital content sitting stale on servers because of a lack of training or understanding of relevance (see also Shivi’s post about Too-Muchness). I know that KA Lite is not some magic quick-fix for education: Amongst many decisive factors, it needs teachers, and those teachers need training; It needs computers, and those computers need electricity.

By allowing ourselves to be honest about these challenges and listening to experience, we can navigate our way to address the root causes of poverty and inequality, by removing barriers to quality education caused by dependency on broadband internet, intellectual property and technological disruption.

Recognizing the threat of complexity and disruption, is especially important for technologists who design and develop software that should have a long-term scope, the life-cycle of which does not include daily automated updates.

It’s ideology!

January 11th marked 5 years since we lost Aaron Swartz, an activist and leader of making information available to those who need it the most. To some, he broke the law, but to me, he sacrificed everything he had for a better world. Sharing information with someone who needs it is a moral imperative, as he put it in the Open Access Manifesto.

I always loved open source as an obvious choice for humanity: Why can’t everyone just share their software inventions freely? If competition is about everyone sitting in their own corner, re-inventing their own wheels, I’m out.

As a side-effect, I caught an allergy for acts of redundant inventing/innovation/disruption hype. I think by now, half my work is about support, community, integrating and helping other people’s contributions, and staying in touch with other people in the field — the other half of my work is software development. And that’s a good thing, because imagine all the re-invented wheels, if we didn’t coordinate our work and sharing our visions, plans and technology across teams and organizations.

Making true open source is an attitude of collaboration: communication and integration with the outside world, not just some legal license.

In order to create true value, we need to highlight the labor of maintenance, and to not get obsessed with innovation. It’s about long-term commitment to keep software functional for the coming generation of computers, and long enough for schools and teachers to learn about the platform, collect experience and integrate it; repeat. These processes are not likely to benefit from technological disruption.

Seeing open source as a principle in aid and education can help to reduce or even eliminate the old power divides between rich and poor countries. Not wanting something back and not retaining ownership is critical to allow for the freedom of a society and its educational institutions to develop.

My work could be said to be in the scope of ICT4D, Information and Communication Technology for Development, a very vibrant and intersecting field, both in academia, amongst NGOs and in the commercial aid sector. I would prefer to argue against this as a constructive term, although it may unify a lot of organizations, people, and projects! Technology is never neutral, and neither is the perception of development. They both quickly become terms that are too broad and overloaded, justifying criticism of manipulation, alternative interests, not addressing fundamental structures that uphold the global inequality, poverty divides etc. Those which we ultimately seek to eliminate as a consequence of our actions of global solidarity and compassion. I prefer to work with education, teacher training, tech support, maintenance and sustainability: The lesser hyped, but classical terms. Nonetheless accepting those true tools in our long struggle towards eliminating poverty and creating equal opportunities for our fellow humans.

Meanwhile . . . at Learning Equality . . .

We develop and share open source, and we contribute to other related open source projects. We are a balanced team of developers, designers, documentation authors, and education specialists.

Technology cannot solve this alone. But creating technology that helps to give access to education and information to those that need it the most is certainly addressing those root causes.

We are not building a prototype for this. We take all types of devices into account and work with a broad scope of education and deployment specialists, and we encourage anyone with an idea or a contribution to participate.

To grow our strength and potential, we are building our own unique organization, making lots of room for discussion, and applying horizontal organizational principles that joins everyone in strategy and visionary level. And I love this.

It’s not about »our« technology

At this point, the Learning Equality development team has started to widen its focus by recognizing our ability to contribute upstream. This is especially true in the Python community, where we interact with lots of other projects, thanks to the affordance of Github.

We’re working towards integrating the offline platform Kolibri (our flagship project) into a range of projects focused on the delivery of educational resources offline (e.g., IIAB, IdeasBox, RACHEL, Kubo, Edulution). Those projects contribute back and help shape the platform in ways that we cannot internally anticipate and design for.

Not least, we are not the sole group of people or organizations: other great platforms such as Kiwix are also addressing offline access and distribution. By participating in their hackathons, we have built many new relations and expanded our horizon, as well as replacing the isolated instinct to compete with an love for collaboration. Together with these teams, we are coordinating protocols, data formats and interoperability.

For the future of Kolibri, I dream that we will manage to polish and engineer our technology all the way until it aligns with the very strict and highly regarded Debian policy and becomes an official open source package for Debian/Ubuntu/Raspbian/Mint etc. Not least, all the other Linux distributions. This way, we can adapt our technology to a finely polished and long-standing line of software that was & is sustainable. State-of-the-art.

We have hundreds of ideas, bugs and rough designs to sift through and prioritize in the coming months, and that makes me think about how this platform, which is open source, does not belong to Learning Equality, but belongs to everyone. We cannot realize all of the ideas alone. That’s why we built a plugin API, that’s why we write documentation, and that’s why we try to make everything as straight-forward and inviting for community to contribute.

If you need me, I’ll be on Github! Thanks for reading, feedback, anecdotes and likewise are most welcome in the comments section :)

First posted on Medium

Blogging in Django: Zinnia or Wagtail-Blog?

Tue, 21 Mar 2017 14:32:26 +0000

Simple answer: If you're already running Wagtail or planning to run it on your site, you'll probably benefit from an integrated blog like wagtail_blog. It may not be full of features in itself, but since Wagtail is, you can easily go where you want. I'm adding support for custom models to wagtail_blog, and after that, the blog system is IMO flexible enough for most subsequent scenarios.

I'm also in the business of migrating an existing Zinnia installation to wagtail_blog, and here's a management command for that: zinnia_to_wagtail.py (Gist).

What about django-blog-zinnia?

This project has been maintained wonderfully for so many years! The codebase is really good (seriously, look at it for inspiration), and it doesn't lack features.

It has so many features! See: django-blog-zinnia.com

It may take a little while to customize its templates and you might want a better visual editor (which is basically what Wagtail+wagtail_blog gives you). But you have all the same flexibility and more with Zinnia. These options may especially come in handy once you start to wanna build out your blog with all those little things. For instance Gravatar support, diversified feeds, pingback, spam checkers, code highlighting etc.

Zinnia also integrates well with django-cms.

Final compromise?

I haven't heard about anyone using custom models in Zinnia to integrate it with Wagtail. But in theory, this is very possible! Without any extensions, just inherit from wagtailcore.models.Page + zinnia.models_bases.entry.AbstractEntry and see where it takes you...

Oh, and btw. this blog is made with wagtail_blog.. it used to be a Wordpress blog.

Suspend using your own custom timing - unlike screen savers

Mon, 13 Mar 2017 07:28:32 +0000

I fail to use automatic standby because:

I often switch off automatic power saving and screen savers and forget to switch them back on
In many cases, it's disturbing unattended processes like downloading
Often defaults to not being active while in docking station or charger

So you might fall asleep in front of a movie. Or you might go out and forget your laptop.

Basically, if I for instance don't respond at 2 AM, I will assume that the computer can suspend because I'm asleep at that time.

Basically, add this script to your desktop user's crontab:

https://gist.github.com/benjaoming/49015c0265f80249da080c41f79ceebc

Skipping it once in a while

You can add your own custom logic in crontab or in the Python script.

A suggestion would be to only allow yourself to switch it off once per run.

For instance, if you want to switch it off in case /tmp/suspend_skip exists: if test -e /tmp/suspend_skip; then rm /tmp/suspend_skip; else python3 awake_or_suspend.py; fi

Debricking OpenWRT on TP Link AC1750 Archer C7 v2.0 via TFTP

This is a little guide for people using Ubuntu who've bricked a router like mine.

References:

Prerequisites:

A workstation running Ubuntu Trusty+ (the tftp server is ancient, so no worries about the Ubuntu version)
A router
Wireshark for debugging

Flashing the router via TFTP

As described in different ways on different sites, but these are the initial steps that will get you started:

Turn off the router
Connect the router's ethernet LAN port 1 to your workstation's ethernet port
Shut down network manager in Ubuntu: sudo service network-manager stop
Open up a command line
Fetch your ethernet card's name by running ifconfig, it's most likely the first on the list, called something like enp0s25.
Configure the card to have a special static IP address that C7 uses to fetch the image: sudo ifconfig enp0s25 192.168.0.66

Now the networking is setup, but there is still no TFTP server.

Inspect router behavior with Wireshark

Before starting a TFTP server, let's instead look at the communication between the router and your workstation.

From command line, fire up Wireshark with root permissions so it can read all the network traffic.

sudo wireshark

Select your network interface by double-clicking it in the Wireshark splash screen menu. Wireshark starts to capture network traffic as soon as this happens, but nothing will happen until we power on the router.

TFTP flash process during router start up

Now that you have Wireshark listening, try turning on the router:

Hold down the "WPS" button next to the ethernet plugs and press the power button. Keep holding it for some seconds until the LEDs stop flashing and only the POWER LED and Ethernet is on.

Look at the Wireshark output. You should see that the router has tried to fetch a TFTP image from 192.168.0.66.

What went wrong tl;dr

Don't feel ashamed if you've bricked a router during your first steps with OpenWRT. It's a lesson. Both for you, and for the sake of improving documentation.

But remember also to blame the companies that manufacture their products to be as difficult as possible to install open source software on.

Let's ask some questions regarding the highly popular TP-Link AC1750 Archer C7 (just nicked C7 ).

Why do they use "v2.0" and yet without documentation in other than serial number prefixing indicate that they've changed the firmware chip? Yes, that's how mine got bricked. One router works fine, the other doesn't.
No external serial port, you have to break warranty to have a serial connection
No plug or indication on circuit board regarding where the serial port is

While TP-Link distributes their products with a paper copy of the GPLv2 license and you can download source code on their website, the company doesn't seem to provide any official embrace of the much superior open source alternative. This would really be in the interest of the consumer and ultimately in the interest of the company itself.

Anyways, I've had it with TP-Link after 3 routers in a row, thinking they were the better choice - up next is Linksys - who do officially embrace open source!

Versioning your Python project for Debian

Sat, 31 Mar 2018 17:43:13 +0000

Version numbers matter! Using a consistent version numbering throughout your project's life matters, too. Changing project name is a high price to pay for cleaning up version errors.

So there's some good news! The Python community's PEP-440 specifies how to use versions.

Here are some examples of versions that abide PEP-440, they are in order!!

0.1.dev1 - Our first release, just playing around. We'll delete it from PyPi.
0.1a1 - After playing around, we decide to release an Alpha 1 for our 0.1 series. We don't know if people might depend on the version, but the Alpha status suggests that the package will be removed from PyPi.
0.1b1 - Then we release a Beta version.
0.1 - Our final release. But of course, more will come.
0.1.post1 - just an hour later, we realize that we forgot a file in MANIFEST.in and we need to release a new version. PyPi does NOT allow us to overwrite versions so we need to bump the number.
0.1.1 - Following a series of issues or fixes, we release a new update.
0.1.2a1 - The next release contains a couple of fixes that we wanna triage first, so we release an alpha.

...you probably get the picture.

If you are in doubt, you can play around with pkg_resource.parse_version like this:

>>> from pkg_resources import parse_version  
>>> parse_version("0.1a1") > parse_version("0.1b1")  
False

Debian compatibility

Debian does not understand "special" characters or words. It just orders by ordinals, letters coming before numbers. You can play around with the rules using this code:

$ if dpkg --compare-versions 0.1a1 lt 0.1; then echo true; else echo false ; fi  
false

Notice that? Debian does not understand the PEP-440 convention. That's because Debian compares 0.1a1 and 0.1 like this: 0=0, 1=1, a>(nothing), 1>(nothing).

In Debian, we should have called it:

$ if dpkg --compare-versions 0.1~a1 lt 0.1; then echo true; else echo false ; fi  
true

This is because ~ orders higher than "nothing".

Maintaining a correct upstream version in Debian?

It's unfortunately not possible to maintain upstream.version~ubuntu1 in case you want to release 0.1a1 followed by0.1.

Debian has introduced "epochs" for fixing wrong or changed versions. However, it will not work to fix our systematic problems. For instance, would you like this?

0:0.1a1
1:0.1
1:0.2a1
2:0.2

Probably not a good idea.

The solution

As stated in the Debian Packaging Policy:

Although this simple approach works most of the times, you may need to adjust package name and upstream version by renaming the upstream source to follow Debian Policy and existing convention.

So with the state of things being that PEP-440 enforces a strict version syntax that's incompatible with a strict Debian version syntax, you should rename your package_VERSION.orig.tar.gz and make sure that VERSION is always translated from your original project's version to Debian's way.

A work flow could look like this:

Bump version number to 0.1a2, release on PyPi
Go to the old debian package source and run uupdate -v 0.1~a2 /path/to/project-0.1a2.tar.gz
Run dch, dpkg-buildpackage -S etc.

So we won't have "pristine upstream version numbers", but we'll get pretty close.

Debating Internet.org

[also brought on FAIR Denmark]

Facebook has launched Internet.org, maybe a solution that will provide internet access to lots of people who need it? In this post, I'll explain what I think about it, because certainly it doesn't seem like a project we can just ignore.

Others are also discussing the issue , and already a group of 65 international organizations have signed a letter with lots of criticism of Facebook's new project. Having just launched Internet.org in Malawi, Lecturer of Blantyre Polytechnic University and a dear contributor to FAIR's projects in Malawi, Bennett Kankuzi, has also joined the growing number of critics.

Common to the criticism, Internet.org is being questioned because it doesn't actually give internet access, and concerns are widespread about the fundamental break-off from net neutrality.

Here's what I found important...

The good

It's with seemingly good reasons and intentions that Internet.org wants to invest in technological transformations to increase internet access, this becomes evident from watching Zuckerberg's 3 minute pitch on the website.

In the video, Zuckerberg doesn't share many details about when or how things will be done, but on the other hand, we get to see a very kind and calm Zuckerberg addressing the needy world. The story wants us to believe that creating Internet.org is easy, just as when he built Facebook in a basement.

Facebook/Internet.org also put out a serious statement on the problems of 4.4 billion people, a very critical aspect of the digital divide. We agree.

As mentioned earlier, a lot of the criticism towards Internet.org has been that it doesn't provide free and unlimited internet access, but rather the opposite: An application for mobile phones giving access to Facebook and partner services. But yet , what we can say is that these services also need a communication infrastructure that's connected to the internet. So regardless of whether or not the end users have real internet access, the project relies on an infrastructure that does.

So far, so good.

And while we're at it, when Facebook/Internet.org talks about open source, they should have credit where credit is due for the Open Compute project. It has actually produced some results, like detailed specs for construction of data centres. Facebook has also succeeded in making prominent players like Apple and Microsoft join their venture.

The bad

Zuckerberg says that we are going to get a 100x more affordable internet by creating a 10x speed boost and making the internet 10x cheaper.

Firstly, there's no credible details supplied for the 10x speed boost. Zuckerberg speaks of compression, caching, and cleaning up airborn signals. Can caching and compression really achieve this and why wouldn't anybody have done that already!? The world works like this: Over-use of data costs money and makes apps run slower. Nobody wants that! It's as if he forgets that bandwidth is also sparse in rich countries and has been even more so in the past! The incentive has always been there and has always been acted upon in both the commercial and academic world. So even if we allow him to have a naive goal, I would say that he's going to go absolutely nowhere with this. Everything that's worth implementing about caching and compression is trivial and will be done where needed.

Cleaning up the airwaves in developing countries? It's hard to believe that this is a serious issue in countries that lack connectivity. I call bullshit here.

But we should also wonder why he never says anything real about the 10x cheaper factor. What does it actually mean? Does it for instance mean that Internet.org will provide a connection that's worth 1/10 of the comparative Western internet connection? As for phones, we know that Chinese manufacturers have already disregarded IP (Intellectual Property) laws and are selling dirt cheap smart phones that are going to be impossible to out-do -- after all, they're made by underpaid Chinese labour, have a minimum of quality and violates all the patents that an open source model was supposed to have delivered a cost-saving on. Again, it seems that we're being fed bullshit.

And why does Zuckerberg talk of making the internet cheaper instead of making people in the "beneficiary" countries richer? Why should they have cheap, low quality products if we could pay everyone on the global market a decent price so everyone could afford quality? Fairphone is a better answer to these problems than trying to make phones for developing countries even cheaper than they already are.

Then Zuckerberg goes to say that making the internet 10x faster and 10x cheaper will make it 100x more affordable. Not only is it hard to understand how this logic works, but ultimately what we get is an internet that's: Less responsive, run on slower devices, and only open up the platform to Facebook and whoever they let through the gate. This is not 100x more affordable. This is something else.

There are also other issues about Internet.org:

Mainly, it's an app and a business model. We should be talking about it in this context.
It isn't open source, yet promotes open source and open content as the solutions to global problems.. how inconsistent is that!?
Even though Facebook said, they'd open up the platform, you have to be approved to get content included, and maybe just maybe they'll let you?
It doesn't support end-to-end encryption, so it's possible for Facebook to read all communication.. again, Facebook and privacy seem to be very distant.

It is far from Net Neutrality, so we have to wonder why Facebook wants this policy in the US and not in their new markets.

And what about open source hardware? Does Nokia and Eriksson really want to offer prime hardware designs for poor people? Or is this just an empty pledge? Certainly the illustration of a soldering irons and mobile phone disassembly followed up by Zuckerberg saying " (...) building low cost open source hardware and phones" would have us believe that Internet.org is going to open source mobile phones.

The ugly

Facebook launched this initiative with the name "Internet.org" as if it compares to other open internet initiatives (hence the .org part). But nowhere can we see that this is an open, democratic initiative. How can you influence it? Where's the source code?

What's also very ugly: If Facebook wanted to use their money to do good, why not join existing initiatives? There's already open source mobile phones, open source operating systems, mobile operators, NGOs addressing the digital divide, business and government initiatives to increase internet access etc.

But this initiative is not about joining the community or unifying civic society, governments, NGOs, and companies... this is about ownership and control of emerging markets and new communication infrastructures.. and one of the methods to get there is this a lobbying / marketing scheme called "Internet.org".

Lastly, let's consider what they say versus what they do in general: We have to compare their good efforts to their actual capacity and reach. For instance, with the Open Compute project with Apple and Microsoft joining: How does the project scale to the full production of the world's top ICT companies? This resembles when NestlÃ© launched a Fairtrade product, a KitKat. Did they take the non-Fairtrade KitKat off the shelves? No. Are they suddenly good guys because of this Fairtrade KitKat, a micro share of their total revenue!? They certainly would want you to think so.

Regenerating Shotwell thumbnails

When reinstalling, upgrading or moving settings, thumbnails may be missing in Shotwell. The problem can look like this:

Each time you import photos, Shotwell will generate thumbnails in two different sizes. If you loose them, you have the problem that they are regenerated on-demand, everytime you scroll by an image. That stinks.

Requirements

You need to do apt-get install sqlite3 imagemagick to fetch requirements.

Source

Here's a script that will regenerate everything, and it's tested on Shotwell 0.18.0. View / fork Gist souce.

#!/bin/bash  
# Based on http://gagor.pl/2014/01/regenerate-thumbnails-in-shotwell-for-last-month/  
THUMB_ROOT=~/.cache/shotwell/thumbs  
# Remove " > date('now','start of month','-1 month')" if you want to re-generate everything  
sqlite3 ~/.local/share/shotwell/data/photo.db \  
  "select id||' '||filename from PhotoTable where date(timestamp,'unixepoch','localtime') > date('now','start of month','-1 month') order by timestamp desc" |  
while read id filename; do  
  tf1=$(printf $THUMB_ROOT/thumbs128/thumb%016x.jpg $id);  
  tf2=$(printf $THUMB_ROOT/thumbs360/thumb%016x.jpg $id);  
  if [ -e "$tf1" ]  
  then  
    echo "Skipping $filename"  
  else  
    echo -n "Generating thumb for $filename";  
    #echo $tf1  
    convert "$filename" -quality 60 -auto-orient -thumbnail 128x128 $tf1  
    convert "$filename" -quality 60 -auto-orient -thumbnail 360x360 $tf2  
    echo  
  fi  
done

Nokia 301 on Wammu / Gammu

I had an older Nokia and got a new one which I needed to upload my phone book to. In order to connect to Nokia 301 from Gammu, plug the phone in with a USB cable and select the "Modem" option on the phone. The others won't work.

Wammu is a GTK frontend to Gammu. You can use for various tasks, but I found that restoring my backup through Wammu failed (even though the file was created with Wammu), while using the commandline tool Gammu directly worked.

Firstly, use the command "gammu-detect" and find the line that reads something with "Nokia", most likely the first line. It will contain something like:

[gammu]  
device = /dev/ttyACM3  
name = Nokia Nokia_301_Dual_SIM  
connection = at

Put this info in your file ~/.gammurc. You then use this command to backup a file created from either Wammu or Gammu:

gammu -s 0 restore ~/Desktop/nokia.backup

The 0 denotes that we are using the 0th section of the configuration file, so if you don't have any other configurations, simply leave the 0.

Ubuntu 14.04 Trusty with DNS server and NetworkManager (disabling dnsmasq)

I've badly been looking for a way to both run an internal network server on eth0 and connecting to any kind of internet device, be it on wlan0 or a USB dongle.

That way, I can be online and browsing documentation, downloading new stuff etc. on the server but not depend on a static configuration but still use Network Manager for its intended purpose.

In /etc/NetworkManager/NetworkManager.conf , uncomment the dnsmasq option like so, because Network Manager's dnsmasq blocks listening to these ports if you want to run your own DNS server:

# dns=dnsmasq

But that's not all! We need /etc/resolv.conf to be updated with the external DNS providers that Network Manager discovers. This can be achieved by removing resolvconf which automatically alters /etc/resolv.conf.

apt-get remove resolvconf

There, done! Now add static configurations in /etc/networks/interfaces.d/ or /etc/networks/interfaces and let Network Manager handle your WLAN interfaces.

Ubuntu, Kickstart, preseeding, and Wireless (WLAN)

I have been trying to get an automated network install running. The starting point is a Kickstart (Kickseed) setup that works fine on LAN. Problem is: Computers do not do PXE netbooting from WLAN, and secondly that the debian installer was not configuring the network correctly. But the steps are very simple.

Get mini.iso, basically the same kernel and initrd that the netbooter is running
Put it on a usb drive to boot the computer (laptop) from: sudo dd if=/path/to/mini.iso of=/dev/[usb device] bs=4096
Make sure that your wireless access point is not protected, just make an open one, we call it UBUNTU-INSTALL
Test that you can connect to it and that the repository is working
Now, boot from the USB drive and press TAB to add boot options: ks=http://1.2.3.4/ks.cfg ksdevice=wlan0 netcfg/wireless_essid=UBUNTU-INSTALL
In order to add your changes to mini.iso, you need to unpack the whole filesystem and edit txt.cfg which is a file that syslinux is using to display the menu.

Changing mini.iso

Tools like UCK do not work for the mini.iso file because it's not using stuff like Casper. So you need to create a custom boot image the hard way:

$ mkdir iso  
$ sudo mount -o loop -t iso9660 mini.iso iso  
$ cp -R iso custom  
$ sudo chmod 600 custom/txt.cfg  
$ sudo nano custom/txt.cfg  
$ sudo mkisofs -r -V "Custom Ubuntu Netboot image" -cache-inodes -J -l -b isolinux.bin -c boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -o custom.iso custom-iso  
$ sudo umount /media/USB-DRIVE # Remember to unmount before writing the usb  
$ sudo dd if=custom.iso of=/dev/[YOU USB DRIVE] bs=4096

Limiting choices in a ModelAdmin list_filter

So, you need to limit choices, because the list of related choices is too long? Then do NOT use limit_choices_to parameter on the ForeignKey! Why? Because you risk having options missing in your forms, deleting relations unknowingly.

For instance, consider that you have the following case, expressed in pseudo code:

ModelClass.related_instance = models.ForeignKey(MyRelatedModel, limit_choices_to = {'active': True}, blank=True, null=True)  
object = <ModelClass instance>  
object.related_instance = <MyRelatedModel instance>  
object.related_instance.active = False

If you edit object, you will see an admin form where <MyRelatedModel instance> is not selectable! Hence, you will risk editing and loosing data without knowing it.

So your best bet is to stop using limit_choices_to and instead define a filter. Luckily, it's quite easy. It's not beautiful, though, since admin.filters.RelatedFieldListFilter does not offer a method to overwrite.

Here is an example:

class RelatedFilter(admin.filters.RelatedFieldListFilter):  
    def __init__(self, *args, **kwargs):  
        super(RelatedFilter, self).__init__(*args, **kwargs)  
        self.lookup_choices = [  
            (x.id, x) for x in  
                models.RelatedModel.objects.filter(status='active')  
        ]





class MyModelAdmin(admin.ModelAdmin):





    list_filter = (('related_field', RelatedFilter), 'other_related_field')

Testing and tracking down Warnings in Django

Warnings are often suppressed through many layers of control mechanisms in Python and Django. However, you should really be aware of these and clean them up once in a while!

In other cases, the Warnings reflect an inconsistent state, such as if you are mixing naive and timezone-aware variables. It can be a real pain to track down RuntimeWarning because it doesn't leave a nice stack trace. The scenario could be that you have assigned a naive DateTime to a field, and the db layer is complaining at runtime.

However, you can run python in a mode where Warning-types raise exceptions with full stack traces. Use the following syntax:

python -W error:"":RuntimeWarning:django.db.models.fields:0 manage.py runserver

'error' means that python should raise an exception - this is what we want!
'RuntimeWarning' means to trigger when such is raised - you can also simply put 'Warning' to trigger for all Warning types.
'django.db.models.fields' means activate for this module (you need the full path). Notice that simply putting 'django' does not activate for all of django, you need to put the specific module or simply '""' to enable for all of Python (which might render some interesting Warning goodies that you have never seen!).
'0' means any line (you probably don't need)

You can read more in Python Documentation chapter: Warning Control.

Django: Log and block brute force attempts

Here is a very simple mechanism for wrapping a decorator around your views to protect them against brute force attempts. For instance, if you have a secret file download available only with the right secret (/view/id/secret-hash/), you expose your view to simple brute force attempts.

Simply put, this decorator will log a 404 response object or Http404 exception, count pr. IP and return status=400 and send you an email whenever a new block is put into place.

Model

class IllegalLookup(LogModifications):  
    """Log and block illegal lookups"""  
    created = models.DateTimeField(  
        verbose_name=_(u'created'),  
        auto_now_add = True,  
    )  
    modified = models.DateTimeField(  
        verbose_name=_(u'created'),  
        auto_now = True,  
    )  
    ip_address = models.CharField(  
        max_length=16,  
        null=True,  
        blank=True,  
        verbose_name=_(u'IP address'),  
    )  
    path = models.CharField(  
        max_length=255,  
        null=True,  
        blank=True,  
        verbose_name=_(u'path'),  
        help_text=_(u'First attempted path is always logged'),  
    )  
    count = models.PositiveIntegerField(  
        default=1,  
    )

    @classmethod  
    def log_lookup(cls, ip_address, path):  
        try:  
            now = timezone.now()  
            expired = now - timedelta(minutes=settings.BLOCK_EXPIRY)  
            lookup = cls.objects.get(ip_address=ip_address,  
                modified__gte=expired)  
            lookup.count += 1  
            lookup.save()  
        except cls.DoesNotExist:  
            # Delete old entries first  
            cls.objects.filter(ip_address=ip_address).delete()  
            lookup = cls.objects.create(ip_address=ip_address,  
                path=path)  
            lookup.save()

    @classmethod  
    def is_blocked(cls, ip_address):  
        try:  
            now = timezone.now()  
            expired = now - timedelta(minutes=settings.BLOCK_EXPIRY)  
            lookup = cls.objects.get(ip_address=ip_address,  
                modified__gte=expired)  
            if lookup.count == settings.BLOCK_ATTEMPTS:  
                mail_admins("IP blocked", "{0} is now blocked, IllegalLookup id: {1}".format(ip_address, lookup.id))  
            if lookup.count > settings.BLOCK_ATTEMPTS:  
                return True  
        except cls.DoesNotExist:  
            pass  
        return False

Decorator

def log_and_block(func):

    def _log_and_block(request, *args, **kwargs):  
        remote_ip = request.META.get('REMOTE_ADDR', None)  
        if IllegalLookup.is_blocked(remote_ip):  
            return HttpResponse('%s is blocked' % remote_ip,  
                status=400)

        is_404 = False  
        is_exception = False  
        try:  
            return_object = func(request, *args, **kwargs)  
            if return_object.status_code == 404:  
                is_404 = True  
        except Http404:  
            is_404 = True  
            is_exception = True

        if is_404:  
            if remote_ip:  
                IllegalLookup.log_lookup(remote_ip,  
                    request.META.get('PATH_INFO', ''))

            if is_exception:  
                raise





        return return_object

    return _log_and_block

Usage

Now, simply wrap the decorator around your view:

@log_and_block  
def my_view(request, id, secret_hash):  
    object = get_object_or_404(models.MyModel, id=id, secret_hash=secret_hash)

Share an internet connection: A nice little script for quickly getting the task done.

This script uses iptable forwarding and dnsmasq to share an internet connection with full relay of remote DNS servers and a local DHCP server. Before trying the script, here is the over all steps:

You connect to the internet in your normal fashion. For instance with a 3G dongle and your network manager applet.
Make sure that nothing is running on port 53, run netstat -tlnp to debug
READ THE SCRIPT before starting and know which interfaces you are using. EXTERNAL is the one connected to the internet and INTERNAL is the one you are sharing the connection via.
Do not let network-manager mange the INTERNAL interface!
You might have to adjust the script, especially the iwconfig part, as different interfaces may have different ways of configuring the WEP key

!/bin/bash

if [ ! whoami = "root" ]
then
echo "Only root"
exit
fi

DHCP='yes'
EXTERNAL=ppp0
INTERNAL=wlan0
INTERNAL_HOST_IP='192.168.10.1'
SHARE_TO_WLAN='yes'
WEPKEY=abe1234567
SSID='mynetwork'
DNSMASQ_CONFIG='dhcp-range=192.168.10.20,192.168.10.255,12h'
DNSMASQ_CONFIG_FILE='/etc/dnsmasq.d/shareconnection'

if [[ $DHCP -eq 'yes' ]]
then

DHCP SERVER (enable below lines)
```
apt-get install dnsmasq  
echo $DNSMASQ_CONFIG > $DNSMASQ_CONFIG_FILE  
    echo "listen-address=192.168.10.1" >> $DNSMASQ_CONFIG_FILE  
    echo interface=$INTERNAL >> $DNSMASQ_CONFIG_FILE  
/etc/init.d/dnsmasq stop
```
fi

echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o $EXTERNAL -j MASQUERADE
/sbin/iptables -A FORWARD -i $EXTERNAL -o $INTERNAL -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i $INTERNAL -o $EXTERNAL -j ACCEPT

ifconfig $INTERNAL down
ifconfig $INTERNAL $INTERNAL_HOST_IP
iwconfig wlan0 mode ad-hoc
iwconfig wlan0 essid $SSID
iwconfig wlan0 key $WEPKEY
ifconfig $INTERNAL up

if [[ $DHCP -eq 'yes' ]]
then
/etc/init.d/dnsmasq start
fi

To Save Everything, Click Here: The Folly of Technological Solutionism

There is a new book coming up with a very remarkable abstract, to be released on March 5, 2013:

In the very near future, â€œsmartâ€ technologies and â€œbig dataâ€ will allow us to make large-scale and sophisticated interventions in politics, culture, and everyday life. Technology will allow us to solve problems in highly original ways and create new incentives to get more people to do the right thing. But how will such â€œsolutionismâ€ affect our society, once deeply political, moral, and irresolvable dilemmas are recast as uncontroversial and easily manageable matters of technological efficiency? What if some such problems are simply vices in disguise? What if some friction in communication is productive and some hypocrisy in politics necessary? The temptation of the digital age is to fix everythingâ€”from crime to corruption to pollution to obesityâ€”by digitally quantifying, tracking, or gamifying behavior. But when we change the motivations for our moral, ethical, and civic behavior we may also change the very nature of that behavior. Technology, Evgeny Morozov proposes, can be a force for improvementâ€”but only if we keep solutionism in check and learn to appreciate the imperfections of liberal democracy. Some of those imperfections are not accidental but by design.

Arguing that we badly need a new, post-Internet way to debate the moral consequences of digital technologies, To Save Everything, Click Here warns against a world of seamless efficiency, where everyone is forced to wear Silicon Valleyâ€™s digital straitjacket.

New Republic has a wonderful chapter from it, Why Social Movements Should Ignore Social Media.