Mailman3 on a VPS using VirtualEnv and and a Third Party SMTP service

Initial Notes

I’ll edit this post as I discover new things, and may not explicitly mark any changes

I installed Mailman3 on a cheap VPS using the instructions at https://docs.mailman3.org/en/latest/install/virtualenv.html. I’m using Mailersend to send the emails, but receiving emails in the server using postfix.

I ran into several difficulties

It helps to have gnu-screen installed so you can switch between a screen in which you’re signed on as mailman and one in which you’re signed on as a standard sudo privileged user

Dependencies

The dependencies should include gcc gettext and . So..

sudo apt install python3-dev python3-venv sassc lynx

should be

sudo apt install python3-dev python3-venv sassc lynx gcc gettext

/bin/bash

On my server, bash isn’t at /usr/bin/bash, it’s just /bin/bash. so instead of ..

sudo useradd -m -d /opt/mailman -s /usr/bin/bash mailman

use ..

sudo useradd -m -d /opt/mailman -s /bin/bash mailman

psycopg2

As the notes suggest, you don’t have to hold back the version of psycopg2-binary anymore. So instead of

(venv)$ pip install wheel mailman psycopg2-binary\<2.9

use

(venv)$ pip install wheel mailman psycopg2-binary

/etc/mailman3/mailman.cfg

The instructions don’t say specifically, but you’ll have to create the /etc/mailman3 folder

Follow instructions that are in the content of the file to create mailman-hyperkitty.cfg. Change the example passwords, keys, and email addresses

Apache2 Mods

I’m using Apache, so the appropriate mods have to be enabled

a2enmod proxy_http a2enmod proxy a2enmod headers

Gunicorn

I’m using Gunicorn. The instructions have you create a file /etc/mailman3/gunicorn.conf but Gunicorn throws a warning about the file name because it wants it to have a python extension.
So add .py to the end of the file name and make it /etc/mailman3/gunicorn.conf.py.

You’ll also have to change the reference to that file in the ExecStart line in /etc/systemd/system/mailmanweb.service

EMAIL_HOST parameters – using a third party smtp service

(This has been extremely frustrating)

I put the following code in /etc/mailman3/settings.py

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'smtp.my_external_smtp_service.com'
EMAIL_PORT = 587
EMAIL_HOST_USER = 'the_user_name_my_smtp_service_gave_me'
EMAIL_HOST_PASSWORD = 'the_password_my_smtp_service_gave_me'
EMAIL_USE_TLS = True

and Mailman3 seemes to ignore it all. But what’s truly frustrating is when I did a test using mailman-web sendtestemail (iaw https://docs.mailman3.org/en/latest/config-web.html), it worked fine. But in operation, if I send an email to an email list, that email won’t be forwarded to list members using those parameters

In retrospect, I understand why. It’s not mailman-web’s job to forward to email lists. That task belongs to mailman core. But for anyone following instructions trying to get this to work, it could be frustrating.

I think those parameters have to remain in settings.py for mailman-web to work (not sure, I’ll test it eventually), but also those parameters go in /etc/mailman3/mailman.cfg as

incoming: mailman.mta.postfix.LMTP
outgoing: mailman.mta.deliver.deliver
lmtp_host: 127.0.0.1
lmtp_port: 8024
smtp_host: smtp.my_external_smtp_service.com
smtp_port: 587
smtp_secure_mode: STARTTLS
smtp_user: the_user_name_my_smtp_service_gave_me
smtp_pass: the_password_my_smtp_service_gave_me

With those parameters in mailman.cfg, it works fine so far

DMARC Mitigation

This may be important to anyone using an external SMTP service

I got to the point where all of the admin messages were going out fine, but the messages to list members were being rejected by my SMPT service, Mailsend.

The problem was I was sending test messages from a domain that wasn’t verified by Mailsend. Which makes sense because the whole world of people who might send a message to the list won’t all have domains verified in my Mailsend account.

The solution is to set DMARC Mitigation in the list settings (as of my writing this, I don’t know of a global setting or a way of changing the default setting, so this has to be done for each list)

This can be done through the web site. In the settings for you list, click ‘DMARC Mitigations’ in your side bar. Pick the ‘Replace..’ or ‘Wrap..’ option – I chose Replace, and click ‘Yes’ for unconditionally.

With this setting, even if someone sends an email from someone@someemail.com, the forwarded message will be from your domain.

I replaced my wife’s smashed iPhone screen

I got my geek on and replaced the screen on my wife’s iPhone 7 Plus. They sell replacement screens on line, some complete with the tools you’ll need, and the internet has plenty of how-to videos. I watched a couple of them (not linking because you have to search for your specific phone)

It took about 2 and a half hours – it’s hard to tell because I got interrupted a couple of times.

I needed good lighting, since my eyes are pushing 60, and I had to borrow my wife’s reading glasses because hers are stronger and mine weren’t up to this task. The screws are about the size of sesame seeds. But I got it done.

In this modern age, try to keep your phone number and don’t lose your phone

At the library where I work stuff like this happens a lot

One case:
– Patron needs help unlocking iPhone
– Tried iTunes but patron also forgot iTunes password
– Sent password reset to Gmail but patron forgot Gmail password
– Gmail sent reset to phone but…

Another case:
– Patron forgot password to bank account
– Bank sent password reset to Gmail, but patron can’t access Gmail
– Gmail sent password reset to phone
– Patron doesn’t have that phone anymore

I wish I can get the word out that your phone is often your last resort to resetting passwords. Try not to lose access to your phone and try to keep your phone number.

If you only remember one password, let it be the one for your phone. If there’s only one thing you can keep from losing, let it be your phone.

Specifying the value format for a Django DateTimeInput using type=”datetime-local”, especially when using inline formsets

TL;DR: For a DateTimeInput widget where type=’datetime-local’, specify the default format to include the T in the middle of the date-time string:

widgets={
   'when':forms.DateTimeInput(
       format='%Y-%m-%dT%H:%M:%S',  
       attrs={'type':'datetime-local'} 
    ),
    ...
}

This was driving me crazy!

I had a model inline formset using a form:

forms.py:

TicketNoteFormset = inlineformset_factory(Ticket, TicketNote, form=TicketNoteForm, extra=10)

The specified form (TicketNoteForm) had a widget specified for a DateTime field:

forms.py:

class TicketTicketNoteForm(forms.ModelForm):
    class Meta:
        model = TicketNote
        fields = [
            'when',
            'text',
        ]
        widgets={
            'when':forms.DateInput(attrs={'type':'date'}),
            'text':forms.TextInput(attrs={'class':'len100'})
        }

My mistake was using DateInput for the field ‘when’, which was a DateTimeField, not a DateField
models.py:

class TicketNote(models.Model):

    when = models.DateTimeField(
        'when',
        default=datetime.now,
        help_text='The date that the note was submitted'
    )

In the HTML forms, the initial value, as expected, was a date-time value but the value attribute of the field was a date value without the time

<input type="date" name="ticketnote_set-0-when" value="2022-01-22 06:52:09" id="id_ticketnote_set-0-when">
<input type="hidden" name="initial-ticketnote_set-0-when" value="2022-01-22 06:52:09" id="initial-ticketnote_set-0-id_ticketnote_set-0-when">
    

The difference between initial and non-initial caused a record to be created even if the form was blank

But it was still broken after I fixed it

Leaving out the widget declaration works, but then I just get a plain text field and I want to take advantage of the browser’s popup calendar

I thought I had the problem fixed by specifying the widget as

forms.py:

DateTimeInput(attrs={'type':'datetime-local'}), 

but that’s still giving me some problems. It worked fine in Firefox, but Chrome ignored the value attribute. So in Chrome

<input type="datetime-local" value="2022-01-01 08:01:00">

displays a blank datetime input and submits an empty string if not updated

I’ll update if I find a good solution.

Update: This works in Firefox and Chrome:
forms.py:

widgets={
   'when':forms.DateTimeInput(
       format='%Y-%m-%dT%H:%M:%S',  
       attrs={'type':'datetime-local'} 
    ),
    ...
}

What Chrome was rejecting was having the value specified without the T between the date portion and the time portion. Django was producing the value without the T. By adding the format argument, I was able to make Django produce a default value that Chrome accepts.

In Django, Using Form to get Field Labels in ListView and DetailView

I don’t know if this is good practice, but to get field labels in a ListView, DetailView, or DeleteView it seems to work fine if you just add a form to the context:

views.py:

class ItemDetailView(DetailView):
    model = Item 
    context_object_name = "item"

    def get_context_data(self, **kwargs):
        context_data = super().get_context_data(**kwargs)
        # just to get the labels
        context_data['form'] = ItemForm
        return context_data

item_detail_view.html:

<div class="label">{{ form.name.label_tag }}</div><div class="value">item.name</div>
<div class="label">{{ form.brand.label_tag }}</div><div class="value">item.brand</div>
<div class="label">{{ form.sn.label_tag }}</div><div class="value">item.sn</div>

You can use label instead of label_tag

This works fine with most fields but fails with ManyToMany fields. Still, it seems to be the easiest way to get most field labels without having to type them in.

Is this a great idea or am I missing some reason that this is bad?

In Django, Effectively Allowing Addition and Removal of Related Models

I’m very new with Django and am happy to accept criticism or suggestions on better ways to do this.

The use case for what I’ve done is this: I’m working on a database to track membership for a small political committee. One of the things we’re interested in is the districts in which our members live. There are different types of districts that we want to track including voting precincts, municipal boroughs, state house districts, state senate districts, and U.S. congressional districts.

For creating or editing a person’s information, there should be a single select box for each type of district. But I want admins to be able to add and remove district types through the admin panel.

To make this work, I have four models: Person, District, DistrictType, and Residency. Residency is an intermediate model for Person and District, but I defined it as a class instead of letting Django create it.

Here is a very simplified version of my models:
Models.py

class DistrictType(models.Model):
    name = models.CharField('Name', max_length=30, help_text='The name of the type of district')

    def __str__(self):
        return self.name


class District(models.Model):
    name = models.CharField('Name', max_length=30, help_text='The name of the type of district')
    district_type = models.ForeignKey(DistrictType, on_delete=models.SET_NULL, null=True)

    def __str__(self):
        return self.name

class Person(models.Model):
    name=models.CharField('Name', max_length=30, help_text='The person's full name'

    def__str__(self):
        Return self.name

class Residency(models.Model):
    district = models.ForeignKey(District, on_delete=models.CASCADE, help_text='The district in which the person lives')
    person = models.ForeignKey(Person, on_delete=models.CASCADE, help_text='The person who lives in the district')

    def __str__(self):
        return self.district + ': ' + self.person

 

I want admin to be able to add and remove districts and district types.

admin.py

from .models import District, DistrictType

class DistrictAdmin(admin.ModelAdmin):
    list_display = ( "name", "district_type")
admin.site.register(District, DistrictAdmin)

class DistrictTypeAdmin(admin.ModelAdmin):
    list_display = ( "name")
admin.site.register(DistrictType, DistrictTypeAdmin)

 

With District and DistricType registered, admins can add and remove district types (and districts) as needed.

I want users to be able to add and edit people without using the admin panel, and I want the person form to have one select box for each type of district. Of course, I have permissions set to restrict who can add and edit people, but I’m not covering that here.

If, for example, I have four types of districts, I want four select boxes on the person form. Each of those select boxes represents a residency. So I have to define a residency form to be used as an inline model form.

forms.py

class PersonForm(ModelForm):
    class Meta:
        model = Person
        fields = (
            "name",
        )

class ResidencyForm(ModelForm):
    district_type_name = forms.CharField(required=False)
    class Meta:
        model = Residency
        fields = (
            "district",
        )

 

Note that district_type_name is not part of the residency model and won’t be used in the update or creation of the object. The reason it is there will be explained below.

The residency model has two fields, person and district, but only district is needed here because person will be taken care of by the formset factory.

views.py

class PersonCreate(CreateView):
    model = Person
    form_class=PersonForm

    district_type_all = DistrictType.objects.all()
    district_type_count = DistrictType.objects.count()

    ResidencyFormSet = inlineformset_factory(Person, Residency, form=ResidencyForm, extra=district_type_count, max_num=district_type_count, fields='__all__')

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)

        residencies = self.ResidencyFormSet( instance = self.object )
        for i in range( self.district_type_count):
            residencies.forms[i].fields['district'].queryset = District.objects.filter(district_type__id=self.district_type_all[i].id)
            residencies.forms[i].fields['district_type_name'].initial = self.district_type_all[i].name
            context['district_label_' + str(i)] = self.district_type_all[i].name
        context['residencies'] = residencies

        return context

    def form_valid(self, form):

        self.object = form.save()

        context = self.get_context_data()

        residencies = context['residencies']
        if residencies.is_valid():
            residencies.save()

        return super(PersonCreate, self).form_valid(form)

 

If you don’t understand inline formsets, I think Daniel Chen’s post will explain their use a lot better than the Django docs.

With the line

ResidencyFormSet = inlineformset_factory(Person, Residency, form=ResidencyForm, extra=district_type_count, max_num=district_type_count, fields='all')
, I created a set of inline forms. The first parameter, Person, defines the parent model. This is why I don’t need a person field in my form definition. The extra parameter defines how many inline forms to display. I want that number to match the amount of district types.

The max_num parameter ensures there will be no more forms displayed than the amount of district types. It’s not necessary in the creation view, although I included it here, but it is necessary in the update view. Without it, there would be four extra select boxes in addition to those that already have data from the previous update or creation.

At this point, I would have four identical inline forms, each with a select box that includes every district in the database. But I want to limit each select box to a certain district type. This is where I think I’m breaking new ground.

Here’s another look at get_context_data in views.py

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)

        residencies = self.ResidencyFormSet( instance = self.object )
        for i in range( self.district_type_count):
            residencies.forms[i].fields['district'].queryset = District.objects.filter(district_type__id=self.district_type_all[i].id)
            residencies.forms[i].fields['district_type_name'].initial = self.district_type_all[i].name
        context['residencies'] = residencies

        return context

 

In get_context_data, I go through each of the residency forms in the formset, and update the queryset for it’s select box. Now each select box is limited to districts of a certain district type.

Here’s where that extra district_type_name field comes in. The default label for the district field is “District”. But on the form, I want each label to be the name of the district type that the select box is limited to. So I populate the unbound field district_type_name with the name of the district type, and use that value as the label.

From person_form.html in my templates directory

    {% csrf_token %}
    <div class="section" >
        Name
        <div class="row" >
            <div class="label" >
                Full Name
            </div >
            <div class="fields" >
                <div class="field" >
                    {{ form.given_name }}
                </div >
            </div >
        </div >
    </div >

    {{ residencies.management_form }}
    <div class="section" >
        Districts
        {% for form in residencies.forms %}
            <div class="row" >
                <div class="label" >
                    {{ form.district_type_name.value }}
                </div >
                <div class="fields" >
                    <div class="field" >
                        {{ form.district }}
                    </div >
                </div >
            </div >
        {% endfor %}
    </div >

    

Here I used district_type_name.value as the label for each select box

Batch File to Maintain Wireless Connection

This is a Windows batch file and a hack for working around intermittent wireless disconnects. If your connection periodically goes to “no internet access”, this script may help. Obviously, it would be better to find and fix the problem but this may keep you going until you do.

This seems so easy that there must be something wrong with it, but I don’t know what that is yet, so I don’t promise anything. It works for me and I can even keep a putty session going with this script running despite frequent intermittent disconnects.

My script figures out the default gateway and the SSID. It pings the default gateway and upon failure, it will reconnect using the SSID. Then it waits 30 seconds and starts over.

I came up with this script after finding plenty of scripts that were pretty close but not quite what I wanted. Some required that you figure out the default gateway and/or the ssid before running the script. Some required the name of the interface. I wanted something that didn’t require my input.

This will open a command window. The command window reports when a reconnection was required and also allows you to hit a key to bypass the 30 second wait period for one round.


@setlocal enableextensions enabledelayedexpansion
@echo off
:loop
for /f "tokens=3 delims= " %%a in ('netsh wlan show interfaces ^| findstr "^....SSID"') do (
set ssid=%%a
)
for /f "tokens=13 delims= " %%a in ('ipconfig ^| findstr "Default.Gateway.*[0-9]"' ) do (
set gateway=%%a
)
ping -n 1 %gateway% | find "TTL="
if errorlevel 1 (
goto :reset
) else (
@timeout /t 30
goto :loop
)
:reset
time /T
netsh wlan connect %ssid%
@timeout /t 60
goto :loop
endlocal


Take a Look at These On Line Scams

This screen popped up after I clicked on a link on Reuters. It looks like an official page from Adobe, telling me I have to install the latest version of Flash. It is not. At the bottom, there is a disclaimer telling us what it is:

We are not affiliated or partnered with Adobe […] This offering is for a download manager that will install independent 3rd party software that will update the advertised program.

I do believe that if I download and run the installer, it will in fact install the latest version of Flash. I’m sure it will also install applications that deliver a steady stream of popup ads. It will probably hijack my browser and prevent me from using Google, instead delivering a bunch of paid-for results whenever I try to search for something. It might do even worse than all that.

But it looks so real. Here’s another example.

This is a page from Sourceforge, a big repository for open source projects, and it’s the Sourceforge page for Xming, a server which allows you to run Linux X applications from a remote server on a Windows desktop. It’s OK if you have no idea what that means. Xming isn’t the problem. The problem is those “Regular Download” and “Premium download” buttons on top. They have nothing to do with Xming and almost nothing to do with Sourceforge. Those are part of an ad. The real download button is the green one closer to the center of the screen. If you click one of the buttons on top, it will take you to another page where you can download another malware installer like the one disguised as the flash updater.

So why Doesn’t Sourceforge do something about these scammer ads on their website? Probably for the same reason I don’t do anything about the ads that may appear on this blog. We don’t see them. In my case, I have nothing at all to do with them. Whatever ads appear on this site are delivered by WordPress, not me. In the case of Sourceforge, they’re just renting space out to Google Ads, and Google Ads is probably working with other companies. Sourceforge has about as much to do with the scammers as your mail carrier does to the scammers who send junk mail to your door.

At any rate, they’re getting trickier out there. They’re doing a good job making their spamware and spyware installers look official, so be sure to double check what you’re clicking on before downloading anything.

My Upgrade to 8.1 is Not Going Smoothly

Update 2013 October 29: And the solution is here: http://blogs.technet.com/b/dennis_schnell/archive/2013/08/31/windows-8-1-wifi-showing-quot-limitied-quot-or-quot-no-internet-access-quot.aspx?PageIndex=2.
To summarize, if you can’t get online after upgrading to 8.1, or if you can’t get online from certain access points, then this might be the solution: Click the link, search for “Kyle”. Read Kyle’s answer then read the paragraph that starts with, “Hey Kyle! You are the man”. Choosing the Broadcom driver worked for me.

Here’s what happened after the original post: As I said in the original post (below), I refreshed the laptop. That dropped me back to Windows 8.0. I waited until I was at school with some free time before trying to upgrade again, so if it failed again I would be able to use a school computer to chat with a tech. It failed again. This time, the tech said, “Unfortunately, the driver for Windows 8.1 isn’t available [from Acer]”, and suggested that I download the latest driver from Broadcom. But I couldn’t find the latest driver on Broadcom’s site. What I found was a forum reply from August

Hi,

Is there a driver that works properly for Windows version 8.1 preview 64 bit.

for BCM57780

Can you please provide me the link to the correct 64 bit driver?

I am always getting disconnects etc

Thank-you,

Shawn

Sorry, no drivers yet for Windows 8.1, they won’t be available until around the time the OS is available at retail. If you’re having problems with the in-box driver then report it to Microsoft so they’re aware of the issue.

Dave

So I used Kyle’s suggestion and seem to be up and running with 8.1.

Below is the original post.

I upgraded from Windows 8 to Windows 8.1 because Windows Store offered the free upgrade in a big purple tile. Then I couldn’t get on-line as school, and I ended up refreshing Windows IAW advice from Acer’s tech support.

I started the upgrade while at school but it took so long that I paused it and finished it at home. Everything seemed fine until I got back to school the next day.

Although I could get online at home, I couldn’t get an IP address from either of the schools’s networks. I tried a few things on my own, asked advice from friends, and then spent an hour or so with Acer in a couple of different sessions over two days. The tech in the last of those sessions gave up and advised me to refresh the laptop. That’s not nearly as bad as wiping your hard disk and reinsalling everything, but it’s still a pain. Refreshing saves your files and re-installs the original configuration plus your Windows Store purchases, but you’re on your own for re-installing any programs you got outside of Windows Store.

The refresh took me back to Windows 8.0, and I will try to upgrade again. I’ll let you know how it goes.

First day with a Win 8 laptop

I ordered and Acer E1 with a third generation (aka Ivy Bridge) Intel i5 processor and Windows 8. It arrived yesterday. Almost certainly, I should have waited a couple of weeks longer as prices are likely to keep falling through August, but ordering the laptop was the only way to stop myself from obsessively checking for deals. If you want a laptop with a 3rd generation processor, the first weeks of August are probably the best time to buy, if you look for deals. My laptop has a 3rd generation i5 processor, 4 gigs of RAM, and a 500GB hard disk, and I got it for $399 plus shipping (after a mail in rebate) from Tiger Direct. It does not have a touchscreen. Prices are falling so dealers can make way for the 4th generation (Haswell) processors. If battery life is important to you, you might want to pay the extra money and hold out for a Haswell.

While looking for deals, don’t get tricked into buying a 2nd generation (Sandy Bridge) chip or older. If you get a great price on an older chip and it’s what you want, that’s fine, as long as you know what you’re getting. The generations are marked by the first number after the dash. My chip is an i5-3230M, and it’s the first “3” that designates the generation. Also don’t get fooled by a “5” after the dash, followed by two more digits instead of three more. Those are actually older.

The i5 seemed like a good match for me. I’m on too much of a budget for an i7 and I’m not a gamer. An i3 is better for budget buyers who don’t run a lot of intensive applications. Budget buyers who will primarily use their laptops for email should also consider the very low prices available on 2nd generation Intel chips, Pentiums, Celerons, and several others out there. AMD makes comparable chips to Intel. The AMD A8-4500M seems to be at about the same level as the Intel i5. As with Intel based computers, be careful about chip model numbers.

One of the first things I tried to do was load Ubuntu 13.04 in a dual boot configuration, but the Ubuntu setup didn’t recognize the existing Windows 8 installation and wanted to format the disk as if it was empty. There’s plenty of information on-line about working around that problem, just like there’s plenty of information about working around the new UEFI security feature, but the less then perfect installation start and the difficulties I’ve read about overcoming the UEFI feature were the last of a dozen or so reasons that made me decide, for now at least, to leave the laptop configured as a single boot Windows 8 machine. Instead, I installed Virtualbox and loaded Ubuntu on a VM.

So I’m succumbing to Microsoft for now. My first impression of Windows 8 is: I like it. The negative backlash against Windows 8 is wrong-headed but I do appreciate the affect that it’s had on prices.

Windows 8 is surprisingly keyboard friendly. Even though it’s designed for touch, I can hit the Windows Key and then type a command or part of a command, like “chrom” or “word”, and get a list of matching applications. It works better than hitting Alt F2 in Unity or Gnome. I can also navigate the start screen easily with the keyboard arrows, and I can shutdown the system without touching the mouse or touchpad by hitting ctrl-alt-delete, then using the tab key to get the power icon. When using the keyboard is easier in Windows then in Linux, it might be time for a shift in thinking.

I also like the live tiles.

I’m still wary of using Windows as my primary OS. I spend a lot of time fixing bugs and removing spyware from my wife and daughter’s computers. My son uses Linux and I never had to fuss with such problems on his laptop. But I like Windows 8 so far. I like the UI more than the UI of previous versions of Windows and I’ve gotten very frustrated with Unity and Gnome. I’m willing to give Windows another go.

So, so far so good. As of Day 1, I’m happy with my Acer and I’m happy with Windows 8.

Update (same day) I’ve already had to remove adware extensions from Chromium. I was getting ads when I clicked on links, and while poking around the settings found an extension called “tidynetworks”. I removed the extension but I don’t yet know if I’ll have to do more to properly get rid of it. I also had something called webcake and I removed that as well.