Tag: IP address

How to install the BIND DNS Server using Webmin, so Asterisk extensions (hopefully) will work even when your Internet connection fails

 

Important
This is an edited version of a post that originally appeared on a blog called The Michigan Telephone Blog, which was written by a friend before he decided to stop blogging. It is reposted with his permission. Comments dated before the year 2013 were originally posted to his blog.

If you run Asterisk you may have encountered this issue: Your Internet connection goes down, and so does your ability to call from extension to extension, even between extensions on your local network. This is a long-standing bug in Asterisk (exactly the sort of bug that drives people to try alternatives such as FreeSWITCH) but let’s say that for whatever reason you need to stick with Asterisk, so you’d like to find a way to make that bug go away.  Without going into all the technical details, the reason that calls fail is that Asterisk can’t access a DNS server.  I’ve read several reports that say the easiest solution is to install the BIND DNS server  on the same machine as your Asterisk server.  If you are also running Webmin on the server, installing and configuring BIND is a relative piece of cake.  So here’s how it’s done.  Please note that most of the images below can be enlarged by clicking on them, and that I have installed the StressFree theme in Webmin, so if it looks a little different from what you’re used to seeing, that’s probably why.

To start with, log into Webmin, click on “Servers”, then click on “BIND DNS Server” (if you don’t find it there, try looking in “Un-used Modules”):

Webmin Servers page — click on "BIND DNS Server"

Assuming you have not previously installed BIND, you’ll get a screen like this.  Just click where it says “Click here”:

Webmin BIND DNS Server error page — click where it says "Click here"

You will then see this screen come up as BIND is installed. Just let it run to completion and (assuming it installs successfully) click on “Return to BIND DNS Server” at the bottom of the page:

Webmin "Install Package" page — click on "Return to BIND DNS Server"

Next, because you don’t yet have an /etc/named.conf file, you’ll see this page.  Click the button for “Setup nameserver for internal non-internet use only” (don’t worry, we’ll fix it in the next steps), then click the bar that says “Create Primary Configuration File and Start Nameserver”:

Webmin — click "Setup nameserver for internal non-internet use only"

At this point BIND is installed and running, but it probably isn’t doing what you want it to, and your system isn’t using it. So the first thing we need to do is tell it where to go when it needs to do a DNS lookup. You should be seeing a page that looks like this — click on “Forwarding and Transfers”:

Webmin BIND DNS Server page — click on "Forwarding and Transfers"

When you get to the following screen, check “Yes” next to “Lookup directly if forwarders cannot?”  You also need to enter one or more addresses of DNS servers that BIND can access when it needs to pull a DNS record.  You might want to give some thought to which DNS servers you want to use, and in what order, before you start entering them. You can enter up to three IP addresses of DNS servers, and then click “Save”. This will throw you out to the previous screen, and if by some chance you want to enter even more DNS servers, you can click on “Forwarding and Transfers” again to come back and enter up to three more servers, until you are finished.  In this example, I have already entered the IP addresses of my router’s DNS Server as the top priority pick,  followed by two Google DNS Server addresses.

Webmin — BIND DNS Server — Forwarding and Transfers page

Once you have done this, you are through configuring BIND directly, but there are two more things we need to do. The first is to make sure that the BIND server starts each time we restart the machine. To do that, go to Webmin’s “System” page and then click on “Bootup and Shutdown”:

Webmin System page — click on "Bootup and Shutdown"

This is a long page so I’m not showing all of it — what you have to do is find the entry for named and check the box next to it:

Webmin Bootup and Shutdown page — check the box next to "named"

Then go to the bottom of the page and click “Start on Boot”:

Bottom of Webmin Bootup and Shutdown page — click "Start on Boot"

At this point BIND is running, and should be using the correct DNS servers, and is set to start at bootup, but your server still isn’t using it for its DNS queries. To get it to do that, go to Webmin’s “Networking” page and click on “Network Configuration”:

Webmin Networking page — click on "Network Configuration"

Once on the Network Configuration page, click on “Hostname and DNS Client”:

Webmin Network Configuration page — click on "Hostname and DNS Client"

Once on the Hostname and DNS Client page, what you need to do is make the first entry in the DNS Servers list 127.0.0.1. If you trust BIND to always be operating, that’s the only entry you need. I didn’t quite trust BIND that much (actually, what I didn’t trust was my ability to set this up correctly) so I set the DNS server in the router as the secondary DNS address. You could use any DNS server as the secondary, or you could choose to just enter the 127.0.0.1 address to use BIND and let it go at that. Personally, I feel a lot more comfortable having a “fallback” DNS. Don’t forget to click “Save” when you are finished making changes here:

Webmin Hostname and DNS Client page - 127.0.0.1 must be first DNS server

That’s all there is to it, as far as I know (if you think I’ve missed anything or done something wrong, the comment section is open!). If you’re like me, the next question you will have is, “How do I know it’s working?” And the easiest way to do that is to go to a Linux command prompt and “dig” some site you have not been to recently twice in a row. Here’s an example, using cnn.com — the part we are interested in is in red:

dig cnn.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_5.3 <<>> cnn.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8274
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 13, ADDITIONAL: 9

;; QUESTION SECTION:
;cnn.com.                       IN      A

;; ANSWER SECTION:
cnn.com.                287     IN      A       157.166.224.25
cnn.com.                287     IN      A       157.166.224.26
cnn.com.                287     IN      A       157.166.226.25
cnn.com.                287     IN      A       157.166.226.26
cnn.com.                287     IN      A       157.166.255.18
cnn.com.                287     IN      A       157.166.255.19

;; AUTHORITY SECTION:
.                       76691   IN      NS      i.root-servers.net.
.                       76691   IN      NS      j.root-servers.net.
.                       76691   IN      NS      k.root-servers.net.
.                       76691   IN      NS      l.root-servers.net.
.                       76691   IN      NS      m.root-servers.net.
.                       76691   IN      NS      a.root-servers.net.
.                       76691   IN      NS      b.root-servers.net.
.                       76691   IN      NS      c.root-servers.net.
.                       76691   IN      NS      d.root-servers.net.
.                       76691   IN      NS      e.root-servers.net.
.                       76691   IN      NS      f.root-servers.net.
.                       76691   IN      NS      g.root-servers.net.
.                       76691   IN      NS      h.root-servers.net.

;; ADDITIONAL SECTION:
b.root-servers.net.     386178  IN      A       192.228.79.201
d.root-servers.net.     402826  IN      A       128.8.10.90
d.root-servers.net.     230000  IN      AAAA    2001:500:2d::d
f.root-servers.net.     370827  IN      A       192.5.5.241
g.root-servers.net.     463754  IN      A       192.112.36.4
h.root-servers.net.     374116  IN      A       128.63.2.53
h.root-servers.net.     517382  IN      AAAA    2001:500:1::803f:235
j.root-servers.net.     185528  IN      A       192.58.128.30
j.root-servers.net.     578747  IN      AAAA    2001:503:c27::2:30

;; Query time: 26 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Sep 16 12:45:41 2011
;; MSG SIZE  rcvd: 512

# dig cnn.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_5.3 <<>> cnn.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8277
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 13, ADDITIONAL: 9

;; QUESTION SECTION:
;cnn.com.                       IN      A

;; ANSWER SECTION:
cnn.com.                223     IN      A       157.166.255.19
cnn.com.                223     IN      A       157.166.224.25
cnn.com.                223     IN      A       157.166.224.26
cnn.com.                223     IN      A       157.166.226.25
cnn.com.                223     IN      A       157.166.226.26
cnn.com.                223     IN      A       157.166.255.18

;; AUTHORITY SECTION:
.                       76627   IN      NS      c.root-servers.net.
.                       76627   IN      NS      d.root-servers.net.
.                       76627   IN      NS      e.root-servers.net.
.                       76627   IN      NS      f.root-servers.net.
.                       76627   IN      NS      g.root-servers.net.
.                       76627   IN      NS      h.root-servers.net.
.                       76627   IN      NS      i.root-servers.net.
.                       76627   IN      NS      j.root-servers.net.
.                       76627   IN      NS      k.root-servers.net.
.                       76627   IN      NS      l.root-servers.net.
.                       76627   IN      NS      m.root-servers.net.
.                       76627   IN      NS      a.root-servers.net.
.                       76627   IN      NS      b.root-servers.net.

;; ADDITIONAL SECTION:
b.root-servers.net.     386114  IN      A       192.228.79.201
d.root-servers.net.     402762  IN      A       128.8.10.90
d.root-servers.net.     229936  IN      AAAA    2001:500:2d::d
f.root-servers.net.     370763  IN      A       192.5.5.241
g.root-servers.net.     463690  IN      A       192.112.36.4
h.root-servers.net.     374052  IN      A       128.63.2.53
h.root-servers.net.     517318  IN      AAAA    2001:500:1::803f:235
j.root-servers.net.     185464  IN      A       192.58.128.30
j.root-servers.net.     578683  IN      AAAA    2001:503:c27::2:30

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Sep 16 12:46:45 2011
;; MSG SIZE  rcvd: 512

Notice how on the first run, it takes 26 msec to do the lookup, because BIND doesn’t have that address cached yet, whereas on the second run it only takes 1 msec to do the lookup!  Could that perhaps improve system performance?  I’ll bet it could! And the SERVER line tells us that it is indeed using our BIND server (127.0.0.1) – if it were using, say, our router’s DNS server then that line would show this:

;; SERVER: 192.168.0.1#53(192.168.0.1)

The idea here is that when your Internet connection takes a dive, Asterisk will still be finding a working DNS server and therefore won’t tank.  That, at least, is the theory I’ve seen on several web sites.  The ONLY thing I am showing here is how to set up BIND using Webmin, and I won’t even guarantee that I’m doing that 100% correctly.  I definitely do not guarantee that it will actually work as intended — you’ll have to test that yourself.  Doing a real test would mean disconnecting your cable or DSL modem, etc. from your router for several hours or days to see if the phones continue to work, and in most households or businesses that idea will go over like a lead balloon.  However, feel free to give it a good test if you like and report the results in the comments.

You may wonder why I selected “Setup nameserver for internal non-internet use only” in the fourth screenshot.  Obviously, that description is not entirely accurate.  The real difference is that if you select that instead of the default “Setup as an internet name server, and download root server information”, it won’t create a “root” DNS zone, which you simply don’t need for this application.  You can use the other option if you want to, but it will download additional information and increase the complexity of your setup.  Either way, you should be able to access the Internet, because we set up DNS forwarding.  If by some chance this BIND server is going to act as a nameserver for your entire network, and you don’t mind the additional traffic and complexity (and it’s the additional traffic that scares me the most, since I have no idea what it’s actually downloading nor how often it’s doing it), then by all means feel free to use the second option.  All I will say is that I used the first. and it works fine, and I’ve seen at least one instance where this same thing is set up using a method other than Webmin, and except for the order of statements it uses an /etc/named.conf file that is identical to what Webmin produces when configured as I have shown here (in other words, no “zones” at all).  I’m just waiting for some Linux purist to say this isn’t the “right” way to do this but keep the goal in mind here — all we are trying to do is work around a bug in Asterisk that should have been fixed years ago, not set up a DNS server to feed an entire subnet.  But again, you can feel free to use whichever of the options you like — it should work either way.

(By the way, if after reading the above you have “setup remorse” — you know, that feeling you get after you’ve installed something that you should have picked a different option — you can get a “do-over” by simply deleting or moving/renaming /etc/named.conf.  If you then exit Webmin’s BIND module and come back in, it should see that named.conf doesn’t exist and start you over at the fourth screen shown above.  Of course, you will lose anything you have already configured from within that module.  If you originally selected the option to download the root server information, I think that’s at least partly stored in the file /etc/db.cache, so you could move or remove that file to make sure it’s not used, however I’m not sure if any other files are or were also downloaded.  That particular file is very small so I’m not worried about that one per se, it’s just that the way things are worded on a couple of pages I read, I don’t know if that’s all it downloads, or if at some point in the middle of the night it rises up and tried to cache all the DNS information for the Internet, or just exactly what it does.  Sometimes I wish people would just give a sentence or two of additional information, so you have a better idea of what’s the right thing to do when you’re setting up something like this.)

Now, if you are a True Linux Geek who somehow stumbled across this article, and are disappointed that it isn’t much more complicated, I’ll refer you to this page.  If you can figure all THAT out, you should be getting paid the big bucks as the networking expert that you are! 🙂

How to isolate a second router from the rest of your local network

 

Important
This is an edited version of a post that originally appeared on a blog called The Michigan Telephone Blog, which was written by a friend before he decided to stop blogging. It is reposted with his permission. Comments dated before the year 2013 were originally posted to his blog.

I was recently asked how to solve a particular problem and I came up with what I think is an interesting solution, especially given my overall rather limited knowledge of networking.  The issue was this: In the home in question, they have cable broadband and a router that feeds jacks throughout the house.  For security reasons, the homeowner never installed any kind of wireless networking (even though his primary router supports it, he keeps it turned off).  Also his primary router is down in the basement.

Recently he got his wife a Motorola XOOM table computer and wouldn’t you know, it requires Wi-Fi access to connect to the Internet.  In order to extend the range, and so that he or his wife could easily turn off the Wi-Fi when the XOOM isn’t in use, he bought a second Wi-Fi router and put it upstairs.  Note that this router is connected BEHIND the original router in the basement.  In other words, the sequence of connection is as follows:

Cable Modem —> Basement (Primary) Router —> Upstairs (Wi-Fi) Router —> Tablet Computer

Now, as I said, he is very security conscious.  So the question he asked me is, if someone managed to break into his Wi-Fi, is there a way to set it up so that they could ONLY get to the Internet, and not to any other system on his local network.  I said I didn’t know, but to first try accessing other machines on his network (the ones that had web interfaces, anyway) from the XOOM.  Turned out that he could do so without any problem.  Because the Wi-Fi router used a different network segment from the original (addresses in the 192.168.2.x range, whereas the original router handed out address in the 192.168.0.x range), as far as anything connected to the Wi-Fi router was concerned, anything on the primary router might as well have been on the Internet (please forgive the non-technical explanation, I’m probably missing several technical details here, but that’s the gist of the problem).

I didn’t think it would be a good idea to try to make the Wi-Fi router use the same address space for both WAN and LAN, and while I could assign it a static IP address on the WAN side, it had to be able to reach the router/gateway at 192.168.0.1.  So here is what we did.

On the PRIMARY router, we took a look at the LAN settings and found that its DHCP server was assigning addresses starting at 192.168.0.2.  We changed that to start at 192.168.0.5 (probably could have used 192.168.0.4 in retrospect).

This way, we could change the WAN address of the Wi-Fi router to use a STATIC IP address of 192.168.0.2, and (this is the important part) a NETMASK of 255.255.255.252.

This means that as far as the Wi-Fi router is concerned, there are only four valid IP addresses in the 192.168.0.x range:

192.168.0.0 (not used)
192.168.0.1 (primary router/gateway)
192.168.0.2 (Wi-Fi router)
192.168.0.3 (Reserved for “broadcast” as far as Wi-Fi router is concerned)

One thing to remember is that after changing the DHCP assignment on the PRIMARY router is that computers already using IP address 192.168.0.2 and 192.168.0.3 will not automatically vacate those addresses until their DHCP lease comes up for renewal.  So if you change the second router’s WAN address to 192.168.0.2, it may not actually be able to connect until the computer or device currently on 192.168.0.2 “loses its lease”.  Rebooting the primary router may help, but in some cases you may have to track down the computer with the conflicting address and shut it off, or if you know how, renew its IP address assignment (this can usually be done from within the network settings panel).  Eventually, though, it should work, and at that point you should find that devices connected to the secondary router cannot connect to any addresses in the 192.168.0.x range outside the three mentioned above, which means they won’t be able to “see” anything else on your network that’s been assigned a DHCP address.

This tip falls into the category of “it worked in this particular situation, but I don’t guarantee it will work for you”.  So if you try this, be sure to test to make sure that the other machines on your primary network are actually unreachable from the secondary router.

Now let the comments begin, telling me how there’s a better way to do this, or why it won’t work, or something to that effect…

Using a dynamic DNS (DDNS) to solve the problem of keeping a firewall open to remote users at changeable IP addresses

 

Important
This is an edited version of a post that originally appeared on a blog called The Michigan Telephone Blog, which was written by a friend before he decided to stop blogging. It is reposted with his permission. Comments dated before the year 2013 were originally posted to his blog.

(Updated July 1, 2011 to include rudimentary test for string returned that doesn’t contain an actual IP address)

One problem faced by Asterisk users (and probably also users of other software PBX’s) is that you want to secure your system by not opening ports up to the entire Internet, but if you have remote users (users not on the same local network as your Asterisk server) you need to make an exception for them to allow them to penetrate your firewall.  If all your external users have fixed IP addresses, it’s not a problem — you simply add a specific rule in your firewall to permit access from each user’s IP address.  However, if their ISP changes their IP address frequently, or if they are using a softphone on a laptop computer, then you can’t just assume they will constantly be at same IP.  And if one of those users happens to be your boss or your mother, they are not going to be happy if they can’t use the phone until they make contact with you, and you enter their new IP address in the firewall.  And they’re probably not going to be real happy if they have to go to a web site or take some other action before they can make and receive calls.

This solution will work for many users in this situation, provided that you are using the iptables firewall. Again, the goal is to keep all your ports closed to outsiders, except for your authorized users. But if you can get each user to set up a Dynamic DNS account and then set their router to do the Dynamic DNS updates (as described here for DD-WRT users), OR failing that if you can get them to install a software Dynamic DNS client on their computer (which is a poorer choice because the computer has to be on for updates to occur), then you can run a script on your Asterisk box every five minutes to check to see if their IP address has changed, and if so, update iptables. I have one script that is called as a cron job every five minutes, and looks like this:

#!/bin/bash
/root/firewall-dynhosts.sh someaddress.afraid.org
/root/firewall-dynhosts.sh someotheraddress.afraid.org
/root/firewall-dynhosts.sh someaddress.no-ip.com

In other words it has one line for each Dynamic DNS host I want to check. For each host it calls a script named firewall-dynhosts.sh which in turn contains this:

#!/bin/bash
# filename: firewall-dynhosts.sh
#
# A script to update iptable records for dynamic dns hosts.
# Written by: Dave Horner (http://dave.thehorners.com)
# Released into public domain.
#
# Run this script in your cron table to update ips.
#
# You might want to put all your dynamic hosts in a sep. chain.
# That way you can easily see what dynamic hosts are trusted.
#
# create the chain in iptables.
# /sbin/iptables -N dynamichosts
# insert the chain into the input chain @ the head of the list.
# /sbin/iptables -I INPUT 1 -j dynamichosts
# flush all the rules in the chain
# /sbin/iptables -F dynamichosts

HOST=$1
HOSTFILE=”/root/dynhosts/host-$HOST”
CHAIN=”dynamichosts” # change this to whatever chain you want.
IPTABLES=”/sbin/iptables”

# check to make sure we have enough args passed.
if [ “${#@}” -ne “1” ]; then
echo “$0 hostname”
echo “You must supply a hostname to update in iptables.”
exit
fi

# lookup host name from dns tables
IP=`/usr/bin/dig +short $HOST | /usr/bin/tail -n 1`
if [ “${#IP}” = “0” ]; then
echo “Couldn’t lookup hostname for $HOST, failed.”
exit
fi

if [ ! `expr “$IP” : ‘([1-9])’` ]; then
echo “Did not return valid IP address, failed.”
exit
fi

OLDIP=””
if [ -a $HOSTFILE ]; then
OLDIP=`cat $HOSTFILE`
# echo “CAT returned: $?”
fi

# has address changed?
if [ “$OLDIP” == “$IP” ]; then
echo “Old and new IP addresses match.”
exit
fi

# save off new ip.
echo $IP>$HOSTFILE

echo “Updating $HOST in iptables.”
if [ “${#OLDIP}” != “0” ]; then
echo “Removing old rule ($OLDIP)”
`$IPTABLES -D $CHAIN -s $OLDIP/32 -j ACCEPT`
fi
echo “Inserting new rule ($IP)”
`$IPTABLES -A $CHAIN -s $IP/32 -j ACCEPT`

echo “Changing rule in /etc/sysconfig/iptables”
sed -i “0,/-A\sdynamichosts\s-s\s$OLDIP\s-j\sACCEPT/s//-A dynamichosts -s $IP -j ACCEPT/” /etc/sysconfig/iptables
# sed -i “s/-A\sdynamichosts\s-s\s$OLDIP\s-j\sACCEPT/-A dynamichosts -s $IP -j ACCEPT/g” /etc/sysconfig/iptables

echo “Sending e-mail notification”
`echo “This is an automated message – please do not reply. The address of dynamic host $HOST has been changed from $OLDIP to $IP. You may need to change the dynamichosts chain in Webmin’s Linux Firewall configuration.” | mail -s “IP address of dynamic host changed on machine name recipient@someaddress.com,anotherrecipient@someaddress.net`

As always, copy and paste the above script, so you can see where the line breaks are really supposed to be (the last line in particular is quite long, and will likely be broken up into four or five lines on the screen). Also, beware of WordPress or other software changing the single or double quotation marks to “prettified” versions — only the plain text normal quotation marks will work.

Note that prior to the first run of the script you will need to run the three commented-out commands shown near the top of the script, right after “create the chain in iptables”, to create the chain. For your convenience here they all are in one place, without the interleaved comment lines:

/sbin/iptables -N dynamichosts
/sbin/iptables -I INPUT 1 -j dynamichosts
/sbin/iptables -F dynamichosts

The lines in blue in firewall-dynhosts.sh are custom additions by me. Just in case something goes wrong, I suggest you make a backup copy of /etc/sysconfig/iptables in a safe place before running this script.  My first addition checks the first character of the string returned in $IP to make sure it is actually a number.  This was a quick and dirty addition to keep it from trying to use a string like ;; connection timed out; no servers could be reached as a valid IP address (yes, it really did that).  I’m sure that the test there could be improved upon (for example, to do a full check for a valid IP address rather than just checking the first digit) but as I say this was a quick and dirty fix.  If you have any suggestions on how to improve it, please leave a comment.  I did find this article, Validating an IP Address in a Bash Script, but it seemed like a bit of overkill considering that in this case what I’m really trying to do is simply weed out error messages.

The second set of additions change the address in the dynamichosts chain of /etc/sysconfig/iptables. Please note that this file may be at a different location in some versions of Linux (such as /etc/iptables.up.rules), if so you will need to change this accordingly. This is particularly important if you run both Webmin and fail2ban. If fail2ban is running it will add some lines to the in-memory version of iptables, so you don’t want to do a simple commit to save the in-memory version back to the iptables file. But at the same time, if you use Webmin’s “Linux Firewall” module to maintain iptables, you want any changes in IP addresses to show up the next time you call up Webmin’s Linux Firewall page. So this simply does a search and replace in /etc/sysconfig/iptables on the rule containing the old IP address, and replaces it with the new one. There are two lines in that section that contain the sed command, the first one will replace only the first instance of the old IP address if it’s in iptables more than once, while the second (which is commented out) would replace all instances of the old IP address. Uncomment whichever you prefer and leave the other commented out, but bear in mind that if two or more of your remote extensions might ever be at the same IP address at the same time, you want the first version (the one that is uncommented above) so that when one of those extensions moves to a different IP address it doesn’t change the IP address for all of the extensions.

Note there’s still a possibility of missing a change if you are actually working in Webmin when a change occurs (since you’ll already have loaded a copy of iptables, and if you then make changes and save it out it could overwrite any change made by the script). But, the last two lines of the script send you an e-mail to alert you to that possibility. If you don’t use Webmin and don’t need or want an e-mail notification for some other reason, you can omit those last two lines, otherwise change the parts in red text to sane values for your situation. While editing, pay attention to the backtick at the end of the line (it’s easy to accidentally delete it when editing an e-mail address — don’t do that!).

When you’re all finished, make sure both scripts are executable and the permissions are correct, then create a cron job to call the first script every five minutes.

The only slight drawback to this method is that when an IP address changes it can take up to ten minutes to update (five for the Dynamic DNS to pick it up, and five more for the cron job to fire that gets it from the Dynamic DNS). Fortunately, most ISP’s tend to change IP address assignments in the middle of the night. Note that using the wrong DNS servers can cause the updates to take significantly longer; I set my computers to use Google’s DNS (8.8.8.8 and 8.8.4.4) and that works fairly well. Note that if ALL your Dynamic DNS addresses are from freedns.afraid.org then you may want to change one line in the above script, from

IP=`/usr/bin/dig +short $HOST | /usr/bin/tail -n 1`

to

IP=`/usr/bin/dig +short @ns1.afraid.org $HOST | /usr/bin/tail -n 1`

This change will specify that the afraid.org DNS server is to be used for these lookups (and ONLY for these lookups, not for every DNS request your system makes – don’t want to overload the servers of this free service!). This may be particularly important if the DNS server you normally use is a caching server that doesn’t always do real-time lookups for each DNS request (for example, if you have installed the BIND DNS Server on your system). If some of the Dynamic DNS addresses come from other services then you could use a similar modification that checks a public DNS service that does not cache entries for long periods of time; as I write this Google’s DNS servers seem to update in near real time.

One thing some may not like is that this script basically hands the “keys to the kingdom” to your authorized users, by giving them access to all ports, or at least all ports not explicitly denied by rules higher in priority. It would be easy enough to change the rule that is written to iptables, or even add additional ones, in the above script, so that you could specify access to individual ports. The other problem is it works great for those external users at fixed locations that don’t move around a lot. It might not work quite as well as well for softphone users on laptops due to the delay between the time they turn on the laptop and the time your Asterisk server picks up the new address.

This has actually worked the best for me of anything I’ve tried so far because once you get the external user’s router set up to do the Dynamic DNS updates, they don’t have to think about doing anything else prior to making a call.

EDIT (December, 2015): If it is not possible or appropriate to update the dynamic DNS automatically from the users’ router, there may be another option. If any of your users have Obihai devices (or possibly another brand of VoIP device that includes an accessible “Auto Provisioning” feature that is not currently being utilized), you may want to know that they do not need to run a separate client to update their dynu.com or freedns.afraid.org dynamic IP address, because an Obihai device (and possibly some other brands of VoIP devices) can do that automatically. This is NOT a recommendation for Obihai devices, but if you or one of your users happens to already have one, here is the information as originally found in this thread on the Obihai forum, posted by user giqcass, who wrote:

Rough Draft for hackish DNS updates:

This hack will let your OBi update Dynamic DNS. It isn’t perfect but it works very well. It’s as simple as calling a url to update the DNS at afraid.org. I believe it would be a simple task to add this feature to the OBi firmware directly. So please add this OBiHai. Pretty please. Until then here you go.

Set up a Dynamic DNS host at http://freedns.afraid.org/
Go to the Dynamic DNS tab.
Copy the “direct” update url link.
Open your Obi admin page.
Click the System management page.
Click Auto Provisioning.
Under “ITSP Provisioning” Change the following.
Method = Periodically
Interval = This setting must be greater then 400 so not to over use resources. I use 3667.
ConfigURL = Paste the update link you got from afraid.org (use http://… not https://…)

Press Submit at the bottom of the page. Restart you OBi.

If you use choose to use dynu.com instead of freedns.afraid.org (which you might because dynu.com doesn’t force you to visit their web site periodically to keep your domain), the procedure is the same (after the first line), except that for the ConfigURL you would use:

http://api.dynu.com/nic/update?hostname=YOUR_DYNU_DYNAMIC_DNS&username=YOUR_DYNU_USERNAME&password=MD5_HASH_OF_PASSWORD

Replace YOUR_DYNU_DYNAMIC_DNS with your dynamic DNS domain name, YOUR_DYNU_USERNAME with the username you use to log into your dynu.com account, and MD5_HASH_OF_PASSWORD with the MD5 hash of your dynu.com password OR your IP Update Password if you have set one (which is recommended). To get the MD5 hash of the password you can enter it on this page. To set or update your IP Update Password, use this page.

The advantage of this is that if one of your users travels and takes their VoIP device with them, it would be able to change the dynamic DNS each time they plug in at a new location (not immediately, but after several minutes at most), so that if you use the technique outlined in this article your server will recognize their current address and permit access. Remember that it’s okay to use more than one Dynamic DNS service simultaneously, in case you or your user are already using a different one that doesn’t provide a simple update URL like dynu.com and freedns.afraid.org do. Other brands of VoIP adapters that have a similar “Auto Provisioning” feature may be able to do this as well, but we don’t have specific information for any of them. If you do, please feel free to add that information in a comment.

Note that we are not recommending any particular free dynamic DNS service. If you want to know what your options are, there is an article on the Best Free Dynamic DNS Services that will show you some options. You want one that is reliable and that will not disappear in a few months, but since we don’t have a crystal ball, we can’t tell you which ones might fit that criteria.

Link: Interesting security technique for Asterisk and FreePBX users (may work with other SIP-based PBX’s also)

 

Important
This is an edited version of a post that originally appeared on a blog called The Michigan Telephone Blog, which was written by a friend before he decided to stop blogging. It is reposted with his permission. Comments dated before the year 2013 were originally posted to his blog.

This article was originally posted in November, 2010.

NOTE: For some reason WordPress absolutely hates it when I try to edit this post, and turns links and other things into piles of steaming poo.  If things don’t look right here please e-mail me or leave a comment and I’ll check it out.  WordPress, I KNOW how I want my articles to look, why can’t you just leave them alone?

One problem faced by some SIP-based VoIP PBX administrators is the issue of security when you have external extensions (that is, extensions located anywhere in the world that’s not a part of your local network). You want to allow those extensions (the ones you’ve authorized) to connect to your system, but you prefer to keep everyone else out, and preferably not even tip them off that there’s a PBX there. The idea is, if the bad guys that would like to break into PBX’s don’t even realize that there is a PBX at your IP address, they won’t waste any time trying to crack into your system.

There have been other suggestions for how to handle this but many of them require your users to take some additional action(s) that they would not normally have to take, and users hate having to lift a finger to do anything to enhance their security. Which brings us to a rather clever technique that doesn’t require user to do anything other than use their phones as they normally would. It might be a tiny bit of a pain to set up initially, but the results may be worth it. I would call this medium level security because if someone is sniffing your packets, this alone may not keep them out, but most of the lowlifes that try to break into PBX’s don’t actually have sufficient access to sniff your packet stream (and also, they’d have to know the exact technique you’re using to be able to crack this). So without further ado…

Secure your VoIP server with the SunshineNetworks knock

(As of October 24, 2012, the above link appears to be DEAD — see the edit at the end of this article)

Note that while the article recommends changing the SIP port to something other than 5060, their basic technique (the “knock”) should still work even if you feel you need to stay on 5060. My only fear about changing the SIP port would be the possibility of losing communications with VoIP providers and with other systems I legitimately send/receive voice traffic to/from. They’re probably going to keep using 5060 even if I don’t. EDIT: My concern here may be unfounded — note the comment below from Alex of Sunshine Networks, who said that “changing the SIP port is quite safe. Your SIP server will send this SIP port along in it’s first SIP invite registration to the VoIP provider. So unless your VoIP provider is actively blocking out anything else than port 5060, it should work fine. We use this technique with 3 different major SIP providers in Australia and never had problems. So far we haven’t seen any unintended consequences.”

I haven’t personally tested this, so if you do, please consider leaving a comment to let me know how it worked for you. The two things I wonder are, do these rules survive a reboot, and can you have more than one secret phrase that would let people in (in case you want to use a different one for each external extension)? EDIT: Those questions are also addressed in Alex’s comment below. Also, those of you running PBX in a Flash should take note of Ward Mundy’s comment about changing an entry in /etc/sysconfig/iptables in this thread. In that same thread, there appears a method to view the “knock” each extension is currently sending — just do “sip debug” from the Asterisk CLI for an hour or so (long enough for all your endpoints to register, after which you can use “sip no debug” to turn it off), then run this at the Linux command prompt (not from the CLI!):

grep "From: " /var/log/asterisk/full|cut -f1 --delimiter=; | sort -u

For each of your remote extensions, you’ll see a line that looks something like this:

From: The Knock <sip:234@nn.nn.nn.nn>

“The Knock” may or may not be enclosed in quotation marks, but it apparently doesn’t matter (you don’t include them in the iptables rules). If you haven’t used a specific “knock”, it could be the actual user’s name, if you set that up when you first set up the endpoint. Anyway, I’d suggest running this BEFORE you actually implement the iptables rules, so you know ahead of time what each endpoint is sending.

EDIT (Added January 8, 2012): I am now using a slight variation on this technique on one of the systems I administer. Without going into too many specifics, I will just note that some SIP devices and VoIP adapters actually already send a unique string that you can use as a “knock” – you do not have to configure a new one, you just need to find out what the device is already sending and use that. For example, let’s say you have an VoIP device connecting to your Asterisk server as extension 234. All you have to do is go to the Asterisk CLI (NOT the Linux command prompt) and enter this:

sip set debug peer 234

(Replace 234 with the actual extension number). Now, assuming that the device is connecting to your server, you will start to see SIP packets scroll across your screen. Within a few minutes you should see one like this (IP addresses have been xx’ed out):

<--- SIP read from UDP:xx.xx.xx.xx:5061 --->

REGISTER sip:xx.xx.xx.xx:5060 SIP/2.0
Call-ID: e10700c2@xx.xx.xx.xx
Content-Length: 0
CSeq: 56790 REGISTER
From: <sip:234@xx.xx.xx.xx>;tag=SP8f427e45f1e19cb24
Max-Forwards: 70
To: <sip:234@xx.xx.xx.xx>
Via: SIP/2.0/UDP xx.xx.xx.xx:5061;branch=z4b9hGK-4f0473a8;rport
Authorization: DIGEST algorithm=MD5,nonce=”37cd169d”,realm=”asterisk”,response=”a726bfed5db321a7bc967b997b5157c2″,uri=”sip:xx.xx.xx.xx:5060″,username=”234″
User-Agent: xxxxxx/xxxxxx-x.x.x.x
Contact: <sip:234@xx.xx.xx.xx:5061>;expires=60;+sip.instance=”<urn:uuid:nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn>”
Allow: ACK,BYE,CANCEL,INFO,INVITE,NOTIFY,OPTIONS,REFER
Supported: replaces

<————->

If you don’t see this you may need to increase the debug level. After you see a packet like this, you can turn off sip debugging:

sip set debug off

The string you are looking for is in the Contact: string above (the nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn is replaced by a unique string). So, where in the instructions for the “knock” they show a sample string such as:

iptables -I door 1 -p udp --dport 5060 -m string --string "mysecretpass" --algo bm -m recent --set --name portisnowopen

I would change the --dport parameter to 5060:5061 (since an VoIP adapter sometimes uses port 5061 for the second service provider — for an device that allows up to fours service providers, use 5060:5063) and the --string parameter to “<urn:uuid:nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn>”, but using the actual string sent by the device, of course. I know the Sunshine Network people recommend using something other than port 5060 but I just can’t bring myself to go quite that far, and even their examples show 5060.

Some other SIP-compliant devices also send unique strings in their REGISTER packets. One that does NOT do so, as far as I am aware, is the venerable Linksys PAP2. And I also do not believe that any of the Sipura line of devices send such a unique string.

Naturally, if an intruder KNOWS you are using that technique, they could try a brute-force attack on the unique string. So I recommend only using this with “uncommon” extension numbers (not 200 or 1000, for example) and with a VERY strong secret/password on the SIP connection. But it is another line of defense against would-be intruders!

EDIT (Added October 24, 2012): The original article, and most of the original site for that matter, seems to have gone offline. While I’m not going to repost the original article here without permission, I will give you a few more details and a couple of excerpts. First, they advised that you change the SIP port to something other than 5060 – they suggested using something in the range 20001 through 49000, though I am not sure why. They uses port 34122 in their examples, and noted that if you are running PBX software that has a “SIP Settings” module, if your find a setting for “Bind Port”, that would be the one to change. Of course if you do this, you then have to change the SIP port on ALL your SIP-based phones and VoIP adapters.

With regard to the “knock” itself, they said this:

Technical information :
… Technically, our knock consists of a secret passphrase which is sent together with the first SIP packet from the phone to the server. SIP packets are text files, very much readable like http packets are. The SIP headers in a REGISTER invite packet have a lot of information, and one of those headers is called the “Display Name”. This display name is used only internally in your Asterisk server and has no other use, so we figured we could fill in anything and the Asterisk functionality would still work fine. We decided to use it as a port knock password.

How does it work :
The Asterisk administrator sets up a simple iptables rule. The iptables rule checks for a secret phrase inside packets sent to the SIP port ( 5060 by default, 34122 after having changed it ). Unless it finds this secret phrase, it will drop the packets to this port. All the remote phone has to do is fill in the “User Name” SIP property on his SIP phone with the secret phrase, and he will be able to connect.

What you then needed to do was to go to into your Asterisk server and from a Linux command prompt, issue the following command:

iptables -N door

Then for EACH “knock” string you want to use, you would do this from the command prompt (note this is only one line, and note that 34122 is the example port and “mykn0ckstr1ng” is an example “knock”):

iptables -I door 1 -p udp –dport 34122 -m string –string “mykn0ckstr1ng” –algo bm -m recent –set –name portisnowopen

If you have anyone that needs to register with your server but cannot send the “knock”, but is at a fixed IP address, you’d add a line like this for each instance (again the port and ip address would probably need to be changed, and note that an entire subnet can be specified as in this example — just leave off the /24 if it’s a single ip address):

iptables -A INPUT -p udp –dport 34122 –source 10.10.1.0/24 -j ACCEPT

Then you would enter these three lines, but again using the correct port rather than 34122. In the first line you see the number 4000 — that is amount of time in seconds that the port will be open, and should be greater than 3600 because that’s the default registration timeout for many sip phones and VoIP adapters. The original article notes that you could use 86400, which is a full day:

iptables -A INPUT -p udp –dport 34122 -m recent –rcheck –seconds 4000 –name portisnowopen -j ACCEPT
iptables -A INPUT -p udp –dport 34122 -j door
iptables -A INPUT -p udp –dport 34122 -j DROP

And finally, to make iptables use these rules, you’d enter:

service iptables save

The original Sunshine Networks article notes that…

This code keeps port 34122 closed ( DROP ) unless someone has opened the door ( door ) in which case they are allowed to pass the door for a little more than 1 hour ( 4000 seconds ). Each time the phone re-registers , the SIP secret pass header is sent, and the door is reopened for 4000 seconds. Since the default SIP reregistration time on many phones is 3600, the 4000 seconds will make sure that as long as the phone is connected to the SIP server, or needs to be connected, the dynamic firewall rule is always active.

Once you have done this, if you configure the Display Name or User Name setting with the “knock” string, it should be able to get through your firewall. Any phone that doesn’t have this string won’t. Of course you can always make the “knock” something that a phone already sends (in a SIP register packet), as noted in the previous edit, and then you don’t have to reconfigure the phone at all. If a phone or device tries to connect without sending the “knock”, the firewall won’t allow it (assuming you haven’t previously created some other rule that allows the traffic to pass) and the connection will fail, or at least that is how it’s supposed to work (I make no guarantees because I didn’t come up with this).

If you enter the command cat /proc/net/ipt_recent/portisnowopen you will get a list of IP addresses that have successfully used the “knock” to connect. Remember that after you implement this, it can take up to an hour for a device to attempt to reconnect.

If anyone ever spots the original article back online, please let me know and I’ll remove this edit. I’d rather you get the information direct from the original source anyway, and the short excerpts I have provided here don’t give the complete overview that the original article provided.

EDIT (February 23, 2014): It appears that there is an archived copy of that original article on the Wayback Machine, although we do not know if it is the most recent edit of that article prior to the site disappearing.

Link: Using IP tables to secure Linux server against common TCP hack attempts

 

Important
This is an edited version of a post that originally appeared on a blog called The Michigan Telephone Blog, which was written by a friend before he decided to stop blogging. It is reposted with his permission. Comments dated before the year 2013 were originally posted to his blog.

This article was originally published in November, 2010.

Iptables
Image by Jordan W via Flickr

I’m not entirely certain of the original source of this article — I found it on one site, but a quick search reveals that the original source is most likely this site, but I may be wrong. The author of that article says he took some of the info in that article (looks like more than “some” from where I sit) from this article: How to: Linux Iptables block common attacks

Related articles found on that site are Using iptables to secure a Linux based Asterisk installation against hack attempts and Securing Asterisk – Fail2Ban (and that latter article looks suspiciously similar to this one: Fail2Ban (with iptables) And Asterisk).

I don’t know how valid or useful any of this is, but if you are running iptables on your system (if you’re not sure enter iptables -V on the command line — it should show you the version of iptables that is installed, if it is installed) then you might want to check these articles out.  And if you find an earlier source for any of these, let me know and I’ll include the links.  I know that in the technical community sometimes information gets copied around, but would it kill you guys to give attribution and a link to the original source when you are lifting information (or even raw text) from someone else’s article?

BETA Perl script for Caller ID popups when using Linksys/Sipura devices

 

Important
This is an edited version of a post that originally appeared on a blog called The Michigan Telephone Blog, which was written by a friend before he decided to stop blogging. It is reposted with his permission. Comments dated before the year 2013 were originally posted to his blog.

Creative Commons License photo credit: bcostin

PLEASE NOTE: This  article has been updated as of December 30, 2008.  This now works with a Mac or Win32 computer (and Linux computers with libnotify installed or readily available, such as those running Ubuntu) and has been updated to reflect that fact. Also, please note that previous versions may have failed on devices/phones with more than two lines – this is (hopefully) fixed as of version 0.7.

If all of the following are true:

You have a Macintosh computer with OS X installed, or a PC with any 32-bit version of Windows installed (basically Windows ’98 through XP), or any version of Linux with libnotify installed

Growl icon
Image via Wikipedia

You have Growl (if you have a Mac) or Snarl (if you have a PC) notifications installed (EDIT: There is now a version of Growl for Windows but at the moment I only have an experimental version of the script for that – see bottom of this page for more information.  It MIGHT work with 32-bit OR 64-bit Windows 7 – feel free to test it).

You have a Linksys or Sipura VoIP adapter on your local subnet or home network and receive calls over it

You would like to see Growl, Snarl or libnotify popups on your computer when a call comes in, showing the caller’s name and number, along with the line that the call came in on and the time and date the call arrived (in case you are out when the call comes in)

You have previously run Perl scripts on your computer, OR are reasonably good at following instructions and problem-solving

AND you are willing to run a script that comes with NO WARRANTY whatsoever (if it breaks, you can keep all the pieces)

Then download this file (now at version 0.92), unzip it and read the Instructions.txt file in the folder appropriate to your computer.

This script is being offered under the GNU General Public License, so if you want to modify it to work on other platforms, you can do that under certain conditions (see the Instructions.txt file for details). Mainly, I’d hope that you’d contribute the modifications back (and please leave a comment on this article if you do that).

I don’t have any kind of regular web page up for this yet, for one thing it’s very rough (very little error-checking) and for another I’m very tired, having spent way too many late nights trying to get this to work. So this post will be more terse than most of my posts, but I think most everything you need to know is in Instructions.txt (and for Mac users, the “How to run at login.rtfd” file) inside the .zip file. Feel free to repost this information to other forums if you think anyone else might be interested.

For those Mac users that wish this were an app: I understand that there is an app called Platypus that allows Perl scripts (and any other types of scripts) to be converted to OS X app bundles. However, what it does not seem to include is any way to specify the command line options, or to load any missing Perl modules. So for now, this script will probably only be usable by those with sufficient knowledge to run a Perl script on their Mac. If I were a bit more knowledgeable, I’d build a preference pane to go in System Preferences, and then have the script read that for its configuration options. But I still have no idea how to make an app install missing Perl modules, particularly when OS X does not come with “make” installed until and unless the Developer Tools are installed (adding something like 3 GB of stuff that is mostly useless to non-developers to your hard drive!).

EDIT: I read somewhere that you can install make without installing the bloated Developer Tools package if you instead install Fink. Then, from a terminal prompt, you can type fink -b install make and supposedly that will do the trick. However, I am told that Fink has not been updated for Snow Leopard, but there is a make package in Rudix that should work with Snow Leopard (mind your paths – Rudix installs make in the /usr/local/bin directory and by default CPAN expects it in /usr/bin, so you may want to adjust the path during CPAN setup, or make a symbolic link in /usr/bin). Since I have not personally tried either of these I have not updated the instructions in the download to reflect this, but if it works you can skip the whole process involved in installing the Developer Tools.

Because this is a Perl script, it lends itself to custom modifications. For example, let’s suppose you have this script running on a Mac, and you are sending Growl notifications to the Mac, but you also have a home theater PC that runs XBMC and/or Boxee, and you’d like to send Caller ID notifications to it as well.  Assuming that Boxee and/or XBMC is configured to allow control via a Web interface, at a fixed IP address and port (192.168.0.150 port 8080 in this example), you could add a line such as this to the script (this is all one line; select and copy to get it all if it gets truncated on your display):

eval {get "http://192.168.0.150:8080/xbmcCmds/xbmcHttp?command=ExecBuiltIn&parameter=XBMC.Notification(Call%20from%20%22$displayname%22%2C$phonenum%20calling%20$lineid[$count]%2C15000%2C%2Fhome%2Fusername%2Fphone.png)"};

The above assumes that you have placed the icon file phone.png (shown at right — right click on the icon and save it) in the user home directory on the destination system (the one running XBMC or Boxee), and that you change ‘username’ to the actual name of the user’s home directory. Note that the icon path requires %2F in place of forward slashes (therefore %2Fhome%2Fusername%2Fphone.png really means /home/username/phone.png) Phone icon - right click and copy imageand this refers to the icon directory and filename — if you choose not to use an icon then leave that part out, along with the %2C that comes just before it.  If you are running XBMC or Boxee on the same system that’s running the script then you should be able to replace 192.168.0.150 with localhost or 127.0.0.1. The above line should be inserted just above the comment line “# Make output string in chosen format” near the end of the Perl script. Keep in mind that this won’t work if you don’t enable control via Web server in XBMC or Boxee, and make sure the port number matches the port in your added line.  Depending on the skin you use, this is generally accomplished by going to Settings, then Network (and in Boxee, then Servers). Then check “Allow control of XBMC via HTTP” (in XBMC) or enable the Web server (in Boxee) and verify the port number is correct.

Starting in Version 0.7 there is a minimal logging function, allowing all detected incoming calls (whether answered or not) to be saved to a text file and/or a comma-quote delimited file. I probably could support other simple formats, but don’t even think about asking for anything more complex (like a rather humorous friend of mine who asked for MySQL integration – considering that he knows how little knowledge I have about Perl programming, and that I have even less knowledge about databases, I’m sure he thought it extremely amusing to make that request). The one thing I really don’t like about offering these scripts in Perl is that it requires the user to know how to install modules from CPAN (or an alternative source if using Win32), but I barely know how to do this stuff in Perl and don’t know any other languages (well, except for QBASIC under MSDOS, but that’s even less compatible across platforms than Perl!).

Starting in Version 0.9 you can use a plain-text file of number-name substitutions, so (for example) if calls from a particular number always display a cryptic Caller ID name, you can change them to say “Uncle Bob” (or some other name if Bob’s not your Uncle, or it’s someone else’s number!). Read the sample config file to see the file formats. Note that the plain text file of number-name substitutions is a separate file, not a section of the optional configuration file, and also note that you must enter the numbers exactly as your VoIP provider sends them (in other words, if they send 8005551234 and you use 18005551234 or 800-555-1234 it will NOT match!).

Starting in Version 0.91 you can use a plain-text file of number-path/file substitutions, so (for example) if calls from a particular number are always from Uncle Bob, you can display Uncle Bob’s picture as the icon whenever a call arrives from that number. Read the sample config file for more information. Note that the plain text file of number-path/file substitutions is a separate file, not a section of the optional configuration file, and also note once again that you must enter the numbers exactly as your VoIP provider sends them.

Version 0.92 sets a rather short timeout on page fetches (still much longer than should be necessary to get the data), in an attempt to resolve a problem where very occasionally the script would just go into a coma, not exiting cleanly but still using memory and CPU cycles, without doing anything useful. I have been running this version for over six months now and have yet to see the script go into a coma, as it often seemed to do in previous versions.

(EDIT added September, 2010:) NOTE regarding EXPERIMENTAL version to work with Growl for Windows.  You should still download the main archive to get the instructions and such, but if you’d prefer to use Growl for Windows rather than Snarl, you can try this experimental version of the script. If you do try it, please let me know if it works as expected (and thanks to Andy Singh for his help with getting this working under Windows 7). Please read the Perl source code to find the module requirements (mentioned on or near line 15 of the script) as they differ slightly from the Snarl version.

Recent Posts

Recent Comments

Archives

Categories

Meta

GiottoPress by Enrique Chavez