adblock: release 1.2.0

* rework download/backup/restore handling
* rework firewall handling
* refine whitelist parsing
* changed default null-ip to 198.18.0.1 to prevent conflicts with bcp38
* spread traffic from ad servers (port 80/443) on two uhttpd instances,
this fixes timeouts on sites with many ssl ad server connections, i.e.
youtube.com
* provide statistics & last runtime for LuCI interface
* documentation update

Signed-off-by: Dirk Brenken <dev@brenken.org>
This commit is contained in:
Dirk Brenken 2016-06-12 18:53:51 +02:00
parent d0eceb6b25
commit 8ab3d8bdc2
6 changed files with 229 additions and 265 deletions

View File

@ -7,7 +7,7 @@
include $(TOPDIR)/rules.mk
PKG_NAME:=adblock
PKG_VERSION:=1.1.17
PKG_VERSION:=1.2.0
PKG_RELEASE:=1
PKG_LICENSE:=GPL-3.0+
PKG_MAINTAINER:=Dirk Brenken <dev@brenken.org>

View File

@ -46,7 +46,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* [zeus tracker](https://zeustracker.abuse.ch)
* => daily updates, approx. 440 entries
* zero-conf like automatic installation & setup, usually no manual changes needed (i.e. ip address, network devices etc.)
* supports a wide range of router modes (incl. AP mode), as long as the firewall and the DNS server are enabled
* supports a wide range of router modes (incl. AP mode), as long as the firewall and the DNS server are enabled & in use
* full IPv4 and IPv6 support
* each blocklist source will be updated and processed separately
* timestamp check to download and process only updated adblock list sources
@ -54,14 +54,13 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* adblock source list parsing by fast & flexible regex rulesets
* additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
* quality checks during & after update of adblock lists to ensure a reliable dnsmasq service
* basic adblock statistics via iptables packet counters
* list states, (overall) list counts & last update time will be stored in uci config
* adblock statistics, last runtime and list states/counts/update times will be stored in uci config for LuCI frontend
* status & error logging to stdout and syslog
* use a dynamic uhttpd instance as an adblock pixel server
* use dynamic iptables rulesets for adblock related redirects/rejects
* use two dynamic uhttpd instances as adblock pixel server, separated for ads delivered on port 80 and on port 443
* use dynamic iptables chains/rulesets for adblock related redirects/rejects
* init system support (start/stop/restart/reload/toggle)
* hotplug support, the adblock start will be triggered by wan 'ifup' event
* adblock toggle to quickly (temporary) switch adblocking 'on' or 'off'
* adblock toggle to quickly switch adblocking 'on' or 'off'
* optional: automatic adblock list backup/restore, backups will be (de-)compressed on the fly (disabled by default)
* optional: add new adblock sources via uci config (see example below)
@ -79,7 +78,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* adblock starts automatically during boot, triggered by wan-ifup event, check _logread -e "adblock"_ for adblock related information
* optional: start/restart/stop the adblock service manually with _/etc/init.d/adblock_
* optional: enable/disable your required adblock list sources in _/etc/config/adblock_ - 'adaway', 'disconnect' and 'yoyo' are enabled by default
* optional: maintain the adblock service in luci under 'System => Startup'
* optional: maintain the adblock service in LuCI under 'System => Startup'
## LuCI adblock companion package
* for easy management of the various blocklist sources and adblock options there is also a nice & efficient LuCI frontend available
@ -102,7 +101,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* **list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
* **new list sources:** you could add new blocklist sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
* **AP mode:** in AP mode adblock uses automatically the local router ip as nullip address. To make sure that your LuCI interface will be still accessible, please change the local uhttpd instance to ports <> 80/443 (see example below)
* **Restricted mode:** to disable flash writes with adblock status information to the adblock config file (used by LuCI frontend), please set 'adb\_restricted' to '1'
* **restricted mode:** to disable flash writes with adblock status information to the adblock config file (used by LuCI frontend), please set 'adb\_restricted' to '1'
* **adblock toggle:** to quickly switch adblocking 'on' or 'off', simply use _/etc/init.d/adblock toggle_
* **configuration update:** to update an outdated adblock config file with the current default version, please run _/etc/init.d/adblock cfgup_, make your individual changes and start the adblock service again
* **debugging:** for script debugging please set the 'adb\_debug' variable in the header of _/etc/init.d/adblock_ to '1'
@ -113,9 +112,10 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* adb\_enabled => main switch to enable/disable adblock service (default: '1', enabled)
* adb\_cfgver => config version string (do not change!) - adblock will check this entry during startup
* adb\_lanif => name of the logical lan interface (default: 'lan')
* adb\_nullport => port of the adblock uhttpd instance (default: '65535')
* adb\_nullipv4 => IPv4 blackhole ip address (default: '192.0.2.1', in AP mode: local router ip)
* adb\_nullipv6 => IPv6 blackhole ip address (default: '::ffff:c000:0201', in AP mode: local router ip)
* adb\_nullport => port of the adblock uhttpd instance used for ads delivered on port 80 (default: '65534')
* adb\_nullportssl => port of the adblock uhttpd instance used for ads delivered on port 443 (default: '65535')
* adb\_nullipv4 => IPv4 blackhole ip address (default: '198.18.0.1', in AP mode: local router ip)
* adb\_nullipv6 => IPv6 blackhole ip address (default: '::ffff:c612:0001', in AP mode: local router ip)
* adb\_forcedns => redirect all local DNS queries to the local dnsmasq resolver (default: '1', enabled)
* adb\_fetchttl => set the timeout for list downloads (default: '5' seconds)
* adb\_restricted => disable updates of the adblock config file (no flash writes) during runtime (default: '0', disabled)
@ -134,26 +134,27 @@ A lot of people already use adblocker plugins within their desktop browsers, but
<pre><code>
ads.example.com
This rule blocks:
This entry blocks the following (sub)domains:
http://ads.example.com/foo.gif
http://server1.ads.example.com/foo.gif
https://ads.example.com:8000/
This rule doesn't block:
This entry does not block:
http://ads.example.com.ua/foo.gif
http://example.com/
</code></pre>
**example whitelist entry (/etc/adblock/adblock.whitelist):**
<pre><code>
analytics.com
here.com
This rule removes _all_ domains from the blocklists with this string in it, i.e.:
google-analytics.com
ssl.google-analytics.com
api.gameanalytics.com
photos.daily-deals.analoganalytics.com
adblockanalytics.com
This entry removes the following (sub)domains from the blocklists:
maps.here.com
here.com
This entry does not remove:
where.com
www.adwhere.com
</code></pre>
**example uhttpd configuration in AP mode:**
@ -171,20 +172,20 @@ config uhttpd 'main'
grep "google-analytics.com" "/tmp/dnsmasq.d/adb_list"*
This will output all matches with corresponding source files:
/tmp/dnsmasq.d/adb_list.winhelp:address=/ssl.google-analytics.com/192.0.2.1
/tmp/dnsmasq.d/adb_list.winhelp:address=/www.google-analytics.com/192.0.2.1
/tmp/dnsmasq.d/adb_list.yoyo:address=/google-analytics.com/192.0.2.1
/tmp/dnsmasq.d/adb_list.winhelp:address=/ssl.google-analytics.com/198.18.0.1
/tmp/dnsmasq.d/adb_list.winhelp:address=/www.google-analytics.com/198.18.0.1
/tmp/dnsmasq.d/adb_list.yoyo:address=/google-analytics.com/198.18.0.1
</code></pre>
**example to find blocked domains on certain sites for whitelisting:**
<pre><code>
1. the easy way ...
enable the network analysis builtins in chrome or firefox to identify domains
which are redirected to the adblock null-ip (default 192.0.2.1), add these domains to your whitelist
which are redirected to the adblock null-ip (default 198.18.0.1), add these domains to your whitelist
2. a bit harder ...
enable 'Log queries' in the dnsmasq configuration (via LuCI Network => DHCP/DNS),
ssh to your router and start tracing with 'logread -f -e "dnsmasq" -e "192.0.2.1"'
ssh to your router and start tracing with 'logread -f -e "dnsmasq" -e "198.18.0.1"'
switch to your client, access the relevant site and check all domains
that are blocked/listed in logread, add these domains to your whitelist
@ -222,10 +223,9 @@ If your awk one-liner works quite well, add a new source section in adblock conf
## Background
This adblock package is a dns/dnsmasq based adblock solution.
Queries to ad/abuse domains are never forwarded and always replied with a local IP address which may be IPv4 or IPv6.
For that purpose adblock uses an ip address from the private 'TEST-NET-1' subnet (192.0.2.1 / ::ffff:c000:0201) by default (in AP mode the local router ip address will be used).
Furthermore all ad/abuse queries will be filtered by ip(6)tables and redirected to internal adblock pixel server (in PREROUTING chain) or rejected (in FORWARD or OUTPUT chain).
All iptables and uhttpd related adblock additions are non-destructive, no hard-coded changes in 'firewall.user', 'uhttpd' config or any other openwrt related config files. There is _no_ adblock background daemon running, the (scheduled) start of the adblock service keeps only the adblock lists up-to-date.
Queries to ad/abuse domains are never forwarded and always replied with a local IP address which may be IPv4 or IPv6. For that purpose adblock uses an ip address from the private 'Benchmark Test' subnet (198.18.0.1 / ::ffff:c612:0001) by default (in AP mode the local router ip address will be used). Furthermore all ad/abuse queries will be filtered by ip(6)tables and redirected to two uhttpd instances, separated for ads delivered on port 80 and on port 443 (in PREROUTING chain) or rejected (in FORWARD or OUTPUT chain).
All iptables and uhttpd related adblock additions are non-destructive, no hard-coded changes in 'firewall.user', 'uhttpd' config or any other system related config files. There is _no_ adblock background daemon running, the (scheduled) start of the adblock service keeps only the adblock lists up-to-date.
## Support
Please join the adblock discussion in this [openwrt forum thread](https://forum.openwrt.org/viewtopic.php?id=59803) or contact me by mail <dev@brenken.org>

View File

@ -32,11 +32,12 @@ f_envload()
# may be overwritten by setting appropriate adblock config options in global section of /etc/config/adblock
#
adb_lanif="lan"
adb_nullport="65535"
adb_nullipv4="192.0.2.1"
adb_nullipv6="::ffff:c000:0201"
adb_nullport="65534"
adb_nullportssl="65535"
adb_nullipv4="198.18.0.1"
adb_nullipv6="::ffff:c612:0001"
adb_whitelist="/etc/adblock/adblock.whitelist"
adb_whitelist_rset="\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\$1)}"
adb_whitelist_rset="\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}"
adb_forcedns=1
adb_fetchttl=5
adb_restricted=0
@ -120,7 +121,7 @@ f_envload()
adb_fetch="$(which wget)"
adb_uci="$(which uci)"
adb_date="$(which date)"
unset adb_srclist adb_revsrclist adb_errsrclist
unset adb_srclist adb_revsrclist
# check 'enabled' & 'version' config options
#
@ -140,26 +141,6 @@ f_envload()
f_exit
fi
# check running dnsmasq instance
#
check="$(pgrep -f "dnsmasq")"
if [ -z "${check}" ]
then
rc=-1
f_log "please enable the local dnsmasq instance to use adblock"
f_exit
fi
# check running firewall
#
check="$(${adb_iptv4} -vnL | grep -F "DROP")"
if [ -z "${check}" ]
then
rc=-1
f_log "please enable the local firewall to use adblock"
f_exit
fi
# get lan ip addresses
#
network_get_ipaddr adb_ipv4 "${adb_lanif}"
@ -189,15 +170,37 @@ f_envload()
then
adb_nullipv4="${adb_ipv4}"
adb_nullipv6="${adb_ipv6}"
if [ "$(${adb_uci} get uhttpd.main.listen_http | grep -Fo "80")" = "80" ] ||
[ "$(${adb_uci} get uhttpd.main.listen_https | grep -Fo "443")" = "443" ]
if [ "$(${adb_uci} -q get uhttpd.main.listen_http | grep -Fo "80")" = "80" ] ||
[ "$(${adb_uci} -q get uhttpd.main.listen_https | grep -Fo "443")" = "443" ]
then
rc=-1
f_log "AP mode detected, set local LuCI instance to ports <> 80/443"
f_exit
elif [ -z "$(pgrep -f "dnsmasq")" ]
then
rc=-1
f_log "please enable the local dnsmasq instance to use adblock"
f_exit
elif [ -z "$(${adb_iptv4} -vnL | grep -Fo "DROP")" ]
then
rc=-1
f_log "please enable the local firewall to use adblock"
f_exit
else
apmode_ok="true"
fi
else
check="$(${adb_uci} -q get bcp38.@bcp38[0].enabled)"
if [ $((check)) -eq 1 ]
then
check="$(${adb_uci} -q get bcp38.@bcp38[0].match | grep -Fo "${adb_nullipv4%.*}")"
if [ -n "${check}" ]
then
rc=-1
f_log "please whitelist '${adb_nullipv4}' in your bcp38 configuration to use default adblock null-ip"
f_exit
fi
fi
fi
# get system release level
@ -349,16 +352,17 @@ f_envcheck()
#
if [ -n "${adb_wanif4}" ]
then
f_firewall "IPv4" "nat" "A" "prerouting_rule" "adb-nat" "-p tcp -d ${adb_nullipv4} -m multiport --dports 80,443 -j DNAT --to-destination ${adb_ipv4}:${adb_nullport}"
f_firewall "IPv4" "filter" "A" "forwarding_rule" "adb-fwd" "-p tcp -d ${adb_nullipv4} -j REJECT --reject-with tcp-reset"
f_firewall "IPv4" "filter" "A" "forwarding_rule" "adb-fwd" "-d ${adb_nullipv4} -j REJECT --reject-with icmp-host-unreachable"
f_firewall "IPv4" "filter" "A" "output_rule" "adb-out" "-p tcp -d ${adb_nullipv4} -j REJECT --reject-with tcp-reset"
f_firewall "IPv4" "filter" "A" "output_rule" "adb-out" "-d ${adb_nullipv4} -j REJECT --reject-with icmp-host-unreachable"
if [ $((adb_forcedns)) -eq 1 ] && [ -n "${adb_landev4}" ]
then
f_firewall "IPv4" "nat" "A" "prerouting_rule" "adb-dns" "-i ${adb_landev4} -p udp --dport 53 -j DNAT --to-destination ${adb_ipv4}:53"
f_firewall "IPv4" "nat" "A" "prerouting_rule" "adb-dns" "-i ${adb_landev4} -p tcp --dport 53 -j DNAT --to-destination ${adb_ipv4}:53"
f_firewall "IPv4" "nat" "prerouting_rule" "prerouting_rule" "0" "dns" "-i ${adb_landev4} -p udp --dport 53 -j DNAT --to-destination ${adb_ipv4}:53"
f_firewall "IPv4" "nat" "prerouting_rule" "prerouting_rule" "0" "dns" "-i ${adb_landev4} -p tcp --dport 53 -j DNAT --to-destination ${adb_ipv4}:53"
fi
f_firewall "IPv4" "nat" "prerouting_rule" "adb-nat" "1" "nat" "-d ${adb_nullipv4} -p tcp --dport 80 -j DNAT --to-destination ${adb_ipv4}:${adb_nullport}"
f_firewall "IPv4" "nat" "prerouting_rule" "adb-nat" "2" "nat" "-d ${adb_nullipv4} -p tcp --dport 443 -j DNAT --to-destination ${adb_ipv4}:${adb_nullportssl}"
f_firewall "IPv4" "filter" "forwarding_rule" "adb-fwd" "1" "fwd" "-d ${adb_nullipv4} -p tcp -j REJECT --reject-with tcp-reset"
f_firewall "IPv4" "filter" "forwarding_rule" "adb-fwd" "2" "fwd" "-d ${adb_nullipv4} -j REJECT --reject-with icmp-host-unreachable"
f_firewall "IPv4" "filter" "output_rule" "adb-out" "1" "out" "-d ${adb_nullipv4} -p tcp -j REJECT --reject-with tcp-reset"
f_firewall "IPv4" "filter" "output_rule" "adb-out" "2" "out" "-d ${adb_nullipv4} -j REJECT --reject-with icmp-host-unreachable"
if [ "${fw_done}" = "true" ]
then
f_log "created volatile IPv4 firewall ruleset"
@ -370,16 +374,17 @@ f_envcheck()
#
if [ -n "${adb_wanif6}" ]
then
f_firewall "IPv6" "nat" "A" "PREROUTING" "adb-nat" "-p tcp -d ${adb_nullipv6} -m multiport --dports 80,443 -j DNAT --to-destination [${adb_ipv6}]:${adb_nullport}"
f_firewall "IPv6" "filter" "A" "forwarding_rule" "adb-fwd" "-p tcp -d ${adb_nullipv6} -j REJECT --reject-with tcp-reset"
f_firewall "IPv6" "filter" "A" "forwarding_rule" "adb-fwd" "-d ${adb_nullipv6} -j REJECT --reject-with icmp6-addr-unreachable"
f_firewall "IPv6" "filter" "A" "output_rule" "adb-out" "-p tcp -d ${adb_nullipv6} -j REJECT --reject-with tcp-reset"
f_firewall "IPv6" "filter" "A" "output_rule" "adb-out" "-d ${adb_nullipv6} -j REJECT --reject-with icmp6-addr-unreachable"
if [ $((adb_forcedns)) -eq 1 ] && [ -n "${adb_landev6}" ]
then
f_firewall "IPv6" "nat" "A" "PREROUTING" "adb-dns" "-i ${adb_landev6} -p udp --dport 53 -j DNAT --to-destination [${adb_ipv6}]:53"
f_firewall "IPv6" "nat" "A" "PREROUTING" "adb-dns" "-i ${adb_landev6} -p tcp --dport 53 -j DNAT --to-destination [${adb_ipv6}]:53"
f_firewall "IPv6" "nat" "PREROUTING" "PREROUTING" "0" "dns" "-i ${adb_landev6} -p udp --dport 53 -j DNAT --to-destination [${adb_ipv6}]:53"
f_firewall "IPv6" "nat" "PREROUTING" "PREROUTING" "0" "dns" "-i ${adb_landev6} -p tcp --dport 53 -j DNAT --to-destination [${adb_ipv6}]:53"
fi
f_firewall "IPv6" "nat" "PREROUTING" "adb-nat" "1" "nat" "-d ${adb_nullipv6} -p tcp --dport 80 -j DNAT --to-destination [${adb_ipv6}]:${adb_nullport}"
f_firewall "IPv6" "nat" "PREROUTING" "adb-nat" "2" "nat" "-d ${adb_nullipv6} -p tcp --dport 443 -j DNAT --to-destination [${adb_ipv6}]:${adb_nullportssl}"
f_firewall "IPv6" "filter" "forwarding_rule" "adb-fwd" "1" "fwd" "-d ${adb_nullipv6} -p tcp -j REJECT --reject-with tcp-reset"
f_firewall "IPv6" "filter" "forwarding_rule" "adb-fwd" "2" "fwd" "-d ${adb_nullipv6} -j REJECT --reject-with icmp6-addr-unreachable"
f_firewall "IPv6" "filter" "output_rule" "adb-out" "1" "out" "-d ${adb_nullipv6} -p tcp -j REJECT --reject-with tcp-reset"
f_firewall "IPv6" "filter" "output_rule" "adb-out" "2" "out" "-d ${adb_nullipv6} -j REJECT --reject-with icmp6-addr-unreachable"
if [ "${fw_done}" = "true" ]
then
f_log "created volatile IPv6 firewall ruleset"
@ -394,22 +399,23 @@ f_envcheck()
then
if [ -n "${adb_wanif4}" ] && [ -n "${adb_wanif6}" ]
then
uhttpd -h "/www/adblock" -k 0 -N 100 -t 0 -T 1 -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullport}" -p "[${adb_ipv6}]:${adb_nullport}"
rc=${?}
uhttpd -h "/www/adblock" -N 25 -T 1 -k 0 -t 0 -R -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullport}" -p "[${adb_ipv6}]:${adb_nullport}"
uhttpd -h "/www/adblock" -N 25 -T 0 -k 0 -t 0 -R -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullportssl}" -p "[${adb_ipv6}]:${adb_nullportssl}"
elif [ -n "${adb_wanif4}" ]
then
uhttpd -h "/www/adblock" -k 0 -N 100 -t 0 -T 1 -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullport}"
rc=${?}
uhttpd -h "/www/adblock" -N 25 -T 1 -k 0 -t 0 -R -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullport}"
uhttpd -h "/www/adblock" -N 25 -T 0 -k 0 -t 0 -R -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullportssl}"
else
uhttpd -h "/www/adblock" -k 0 -N 100 -t 0 -T 1 -D -S -E "/index.html" -p "[${adb_ipv6}]:${adb_nullport}"
rc=${?}
uhttpd -h "/www/adblock" -N 25 -T 1 -k 0 -t 0 -R -D -S -E "/index.html" -p "[${adb_ipv6}]:${adb_nullport}"
uhttpd -h "/www/adblock" -N 25 -T 0 -k 0 -t 0 -R -D -S -E "/index.html" -p "[${adb_ipv6}]:${adb_nullportssl}"
fi
rc=${?}
if [ $((rc)) -eq 0 ]
then
f_log "created volatile uhttpd instance"
f_log "created volatile uhttpd instances"
else
f_log "failed to initialize volatile uhttpd instance" "${rc}"
f_restore
f_log "failed to initialize volatile uhttpd instances" "${rc}"
f_exit
fi
fi
@ -446,26 +452,44 @@ f_depend()
f_firewall()
{
local ipt="${adb_iptv4}"
local nullip="${adb_nullipv4}"
local proto="${1}"
local table="${2}"
local ctype="${3}"
local chsrc="${3}"
local chain="${4}"
local notes="${5}"
local rules="${6}"
local chpos="${5}"
local notes="adb-${6}"
local rules="${7}"
# select appropriate iptables executable for IPv6
#
if [ "${proto}" = "IPv6" ]
then
ipt="${adb_iptv6}"
nullip="${adb_nullipv6}"
fi
# check whether iptables rule already applied and proceed accordingly
# check whether iptables chain already exist
#
rc="$("${ipt}" -w -t "${table}" -C "${chain}" -m comment --comment "${notes}" ${rules}; printf ${?})"
rc="$("${ipt}" -w -t "${table}" -nL "${chain}" >/dev/null 2>&1; printf ${?})"
if [ $((rc)) -ne 0 ]
then
"${ipt}" -w -t "${table}" -"${ctype}" "${chain}" -m comment --comment "${notes}" ${rules}
"${ipt}" -w -t "${table}" -N "${chain}"
"${ipt}" -w -t "${table}" -A "${chain}" -m comment --comment "${notes}" -j RETURN
"${ipt}" -w -t "${table}" -A "${chsrc}" -d "${nullip}" -m comment --comment "${notes}" -j "${chain}"
fi
# check whether iptables rule already exist
#
rc="$("${ipt}" -w -t "${table}" -C "${chain}" -m comment --comment "${notes}" ${rules} >/dev/null 2>&1; printf ${?})"
if [ $((rc)) -ne 0 ]
then
if [ $((chpos)) -eq 0 ]
then
"${ipt}" -w -t "${table}" -A "${chain}" -m comment --comment "${notes}" ${rules}
else
"${ipt}" -w -t "${table}" -I "${chain}" "${chpos}" -m comment --comment "${notes}" ${rules}
fi
rc=${?}
if [ $((rc)) -eq 0 ]
then
@ -552,91 +576,21 @@ f_rmconfig()
{
local src_name
local rm_done="${1}"
local restore_done="${2:-false}"
for src_name in ${rm_done}
do
src_name="${src_name#*.}"
if [ "${restore_done}" = "true" ]
then
src_name="${src_name%.*}"
"${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=list restored"
else
"${adb_uci}" -q delete "adblock.${src_name}.adb_src_count"
"${adb_uci}" -q delete "adblock.${src_name}.adb_src_timestamp"
fi
"${adb_uci}" -q delete "adblock.${src_name}.adb_src_count"
"${adb_uci}" -q delete "adblock.${src_name}.adb_src_timestamp"
done
}
# f_restore: restore last adblock list backups and restart dnsmasq
#
f_restore()
{
local rm_done
local restore_done
# remove bogus adblock lists
#
if [ -n "${adb_revsrclist}" ]
then
rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_revsrclist} \) -print -exec rm -f "{}" \;)"
rc=${?}
if [ $((rc)) -eq 0 ] && [ -n "${rm_done}" ]
then
f_rmconfig "${rm_done}"
f_log "all bogus adblock lists removed"
elif [ $((rc)) -ne 0 ]
then
f_log "error during removal of bogus adblock lists" "${rc}"
f_exit
fi
fi
# restore backups
#
if [ "${backup_ok}" = "true" ]
then
restore_done="$(find "${adb_dir_backup}" -maxdepth 1 -type f -name "${adb_dnsprefix}*.gz" -print -exec cp -pf "{}" "${adb_dnsdir}" \;)"
rc=${?}
if [ $((rc)) -eq 0 ] && [ -n "${restore_done}" ]
then
find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*.gz" -exec gunzip -f "{}" \;
f_log "all available backups restored"
elif [ $((rc)) -ne 0 ] && [ -n "${restore_done}" ]
then
f_log "error during restore of adblock lists" "${rc}"
f_exit
fi
else
f_log "backup service disabled, nothing to restore"
fi
# (re-)try dnsmasq restart without bogus adblock lists / with backups
#
if [ -n "${restore_done}" ] || [ -n "${rm_done}" ]
then
/etc/init.d/dnsmasq restart
sleep 1
check="$(pgrep -f "dnsmasq")"
if [ -n "${check}" ]
then
rc=0
f_cntconfig
f_log "adblock lists with overall ${adb_count} domains loaded"
else
rc=120
f_log "dnsmasq restart failed, please check 'logread' output" "${rc}"
fi
fi
f_exit
}
# f_exit: delete (temporary) files, generate statistics and exit
#
f_exit()
{
local ipv4_adblock=0
local ipv6_adblock=0
local ipv4_blk=0 ipv4_all=0 ipv4_pct=0
local ipv6_blk=0 ipv6_all=0 ipv6_pct=0
local lastrun="$(${adb_date} "+%d.%m.%Y %H:%M:%S")"
# delete temporary files & directories
@ -656,19 +610,26 @@ f_exit()
then
if [ -n "${adb_wanif4}" ]
then
ipv4_adblock="$(${adb_iptv4} -t nat -vnL | awk '$11 ~ /^adb-nat$/ {sum += $1} END {printf sum}')"
ipv4_adblock="$((${ipv4_adblock} + $(${adb_iptv4} -vnL | awk '$11 ~ /^adb-(fwd|out)$/ {sum += $1} END {printf sum}')))"
ipv4_blk="$(${adb_iptv4} -t nat -vnL adb-nat | awk '$3 ~ /^DNAT$/ {sum += $1} END {printf sum}')"
ipv4_all="$(${adb_iptv4} -t nat -vnL PREROUTING | awk '$3 ~ /^prerouting_rule$/ {sum += $1} END {printf sum}')"
if [ $((ipv4_all)) -gt 0 ] && [ $((ipv4_blk)) -gt 0 ] && [ $((ipv4_all)) -gt $((ipv4_blk)) ]
then
ipv4_pct="$(printf "${ipv4_blk}" | awk -v all="${ipv4_all}" '{printf( "%5.2f\n",$1/all*100)}')"
fi
fi
if [ -n "${adb_wanif6}" ]
then
ipv6_adblock="$(${adb_iptv6} -t nat -vnL | awk '$10 ~ /^adb-nat$/ {sum += $1} END {printf sum}')"
ipv6_adblock="$((${ipv6_adblock} + $(${adb_iptv6} -vnL | awk '$10 ~ /^adb-(fwd|out)$/ {sum += $1} END {printf sum}')))"
ipv6_blk="$(${adb_iptv6} -t nat -vnL adb-nat | awk '$3 ~ /^DNAT$/ {sum += $1} END {printf sum}')"
ipv6_all="$(${adb_iptv6} -t nat -vnL PREROUTING | awk '$3 ~ /^(adb-nat|DNAT)$/ {sum += $1} END {printf sum}')"
if [ $((ipv6_all)) -gt 0 ] && [ $((ipv6_blk)) -gt 0 ] && [ $((ipv6_all)) -gt $((ipv6_blk)) ]
then
ipv6_pct="$(printf "${ipv6_blk}" | awk -v all="${ipv6_all}" '{printf( "%5.2f\n",$1/all*100)}')"
fi
fi
if [ -n "$(${adb_uci} -q changes adblock)" ]
then
"${adb_uci}" -q commit "adblock"
fi
f_log "firewall statistics (IPv4/IPv6): ${ipv4_adblock}/${ipv6_adblock} ad related packets blocked"
"${adb_uci}" -q set "adblock.global.adb_percentage=${ipv4_pct}%/${ipv6_pct}%"
"${adb_uci}" -q set "adblock.global.adb_lastrun=${lastrun}"
"${adb_uci}" -q commit "adblock"
f_log "firewall statistics (IPv4/IPv6): ${ipv4_pct}%/${ipv6_pct}% of all packets in prerouting chain are ad related & blocked"
f_log "domain adblock processing finished successfully (${adb_scriptver}, ${adb_sysver}, ${lastrun})"
elif [ $((rc)) -gt 0 ]
then

View File

@ -28,8 +28,8 @@ fi
# get current directory and set script/config version
#
adb_scriptdir="${0%/*}"
adb_scriptver="1.1.17"
adb_mincfgver="2.1"
adb_scriptver="1.2.0"
adb_mincfgver="2.2"
# source in adblock function library
#
@ -88,7 +88,7 @@ do
adb_srclist="${adb_srclist} -a ! -name ${adb_dnsprefix}.${src_name}*"
fi
# only download adblock list with newer/updated timestamp
# download only block list with newer/updated timestamp
#
if [ "${src_name}" = "blacklist" ]
then
@ -99,7 +99,7 @@ do
if [ -z "${url_time}" ]
then
url_time="$(date)"
f_log " no online timestamp received, current date will be used"
f_log " online timestamp not received, current date will be used"
fi
if [ -z "${list_time}" ] || [ "${list_time}" != "${url_time}" ] || [ ! -r "${adb_dnsfile}" ] ||\
([ "${backup_ok}" = "true" ] && [ ! -r "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" ])
@ -107,7 +107,6 @@ do
if [ "${src_name}" = "blacklist" ]
then
tmp_domains="$(cat "${url}")"
rc=${?}
elif [ "${src_name}" = "shalla" ]
then
shalla_archive="${adb_tmpdir}/shallalist.tar.gz"
@ -127,30 +126,31 @@ do
break
fi
done
rm -f "${shalla_archive}"
rm -rf "${adb_tmpdir}/BL"
tmp_domains="$(cat "${shalla_file}")"
rc=${?}
rm -rf "${adb_tmpdir}/BL"
rm -f "${shalla_archive}"
rm -f "${shalla_file}"
fi
else
tmp_domains="$(${adb_fetch} ${fetch_parm} --output-document=- "${url}")"
rc=${?}
fi
rc=${?}
else
f_log " source doesn't change, no update required"
continue
fi
# check download result and prepare domain output by regex patterns
# check download result and prepare domain output, backup/restore if needed
#
if [ $((rc)) -eq 0 ] && [ -n "${tmp_domains}" ]
then
count="$(printf "%s\n" "${tmp_domains}" | awk "${src_rset}" | tee "${adb_tmpfile}" | wc -l)"
f_log " source download finished (${count} entries)"
if [ "${src_name}" = "shalla" ]
"${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=${url_time}"
if [ "${backup_ok}" = "true" ]
then
rm -f "${shalla_file}"
gzip -cf "${adb_tmpfile}" > "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz"
fi
f_log " source download finished (${count} entries)"
unset tmp_domains
elif [ $((rc)) -eq 0 ] && [ -z "${tmp_domains}" ]
then
@ -159,53 +159,61 @@ do
continue
else
rc=0
if [ -z "${adb_errsrclist}" ]
if [ "${backup_ok}" = "true" ] && [ -r "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" ]
then
adb_errsrclist="-name ${adb_dnsprefix}.${src_name}.gz"
gunzip -cf "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" > "${adb_tmpfile}"
count="$(wc -l < "${adb_tmpfile}")"
"${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=list restored"
f_log " source download failed, list restored (${count} entries)"
else
adb_errsrclist="${adb_errsrclist} -o -name ${adb_dnsprefix}.${src_name}.gz"
"${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=download failed"
f_log " source download failed"
continue
fi
"${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=download failed"
f_log " source download failed"
continue
fi
# remove whitelist domains, sort domains and make them unique,
# finally rewrite ad/abuse domain information to separate dnsmasq files
# rewrite ad/abuse domain information to separate dnsmasq files
#
if [ $((count)) -gt 0 ] && [ -n "${adb_tmpfile}" ]
then
if [ -s "${adb_tmpdir}/tmp.whitelist" ]
then
grep -vf "${adb_tmpdir}/tmp.whitelist" "${adb_tmpfile}" | sort -u | eval "${adb_dnsformat}" > "${adb_dnsfile}"
rc=${?}
else
sort -u "${adb_tmpfile}" | eval "${adb_dnsformat}" > "${adb_dnsfile}"
rc=${?}
fi
rc=${?}
# prepare find statement with revised adblock list sources
#
if [ -z "${adb_revsrclist}" ]
then
adb_revsrclist="-name ${adb_dnsprefix}.${src_name}"
else
adb_revsrclist="${adb_revsrclist} -o -name ${adb_dnsprefix}.${src_name}"
fi
# store source timestamp in config
# finish domain processing, prepare find statement with revised adblock list source
#
if [ $((rc)) -eq 0 ]
then
"${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=${url_time}"
if [ -z "${adb_revsrclist}" ]
then
adb_revsrclist="-name ${adb_dnsprefix}.${src_name}"
else
adb_revsrclist="${adb_revsrclist} -o -name ${adb_dnsprefix}.${src_name}"
fi
f_log " domain merging finished"
else
f_log " domain merging failed" "${rc}"
f_restore
rc=0
rm -f "${adb_dnsfile}"
if [ "${backup_ok}" = "true" ] && [ -r "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" ]
then
rm -f "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz"
fi
f_log " domain merging failed, list removed"
continue
fi
else
rm -f "${adb_dnsfile}"
if [ "${backup_ok}" = "true" ] && [ -r "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" ]
then
rm -f "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz"
fi
"${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=empty domain input"
f_log " empty domain input received"
f_log " empty domain input received, list removed"
continue
fi
done
@ -215,45 +223,21 @@ done
if [ -n "${adb_srclist}" ]
then
rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_srclist} \) -print -exec rm -f "{}" \;)"
rc=${?}
if [ "${backup_ok}" = "true" ] && [ -n "${rm_done}" ]
then
find "${adb_dir_backup}" -maxdepth 1 -type f \( ${adb_srclist} \) -exec rm -f "{}" \;
fi
else
rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -print -exec rm -f "{}" \;)"
rc=${?}
if [ "${backup_ok}" = "true" ]
then
find "${adb_dir_backup}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -exec rm -f "{}" \;
fi
fi
if [ $((rc)) -eq 0 ] && [ -n "${rm_done}" ]
if [ -n "${rm_done}" ]
then
f_rmconfig "${rm_done}"
f_log "remove disabled adblock lists"
elif [ $((rc)) -ne 0 ] && [ -n "${rm_done}" ]
then
f_log "error during removal of disabled adblock lists" "${rc}"
f_exit
fi
# partial restore of adblock lists in case of download errors
#
if [ "${backup_ok}" = "true" ] && [ -n "${adb_errsrclist}" ]
then
restore_done="$(find "${adb_dir_backup}" -maxdepth 1 -type f \( ${adb_errsrclist} \) -print -exec cp -pf "{}" "${adb_dnsdir}" \;)"
rc=${?}
if [ $((rc)) -eq 0 ] && [ -n "${restore_done}" ]
then
find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*.gz" -exec gunzip -f "{}" \;
f_rmconfig "${restore_done}" "true"
f_log "partial restore done"
elif [ $((rc)) -ne 0 ]
then
f_log "error during partial restore" "${rc}"
f_exit
fi
f_log "disabled adblock lists removed"
fi
# make separate adblock lists entries unique
@ -289,7 +273,7 @@ fi
#
f_cntconfig
adb_count="$(${adb_uci} -q get "adblock.global.adb_overall_count")"
if [ -n "${adb_revsrclist}" ] || [ -n "${rm_done}" ] || [ -n "${restore_done}" ] || [ -n "${mv_done}" ]
if [ -n "${adb_revsrclist}" ] || [ -n "${rm_done}" ]
then
"${adb_uci}" -q set "adblock.global.adb_dnstoggle=on"
/etc/init.d/dnsmasq restart
@ -299,31 +283,30 @@ then
then
f_log "adblock lists with overall ${adb_count} domains loaded"
else
rc=100
f_log "dnsmasq restart failed, please check 'logread' output" "${rc}"
f_restore
f_log "dnsmasq restart failed, retry without newly generated block lists"
rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_revsrclist} \) -print -exec rm -f "{}" \;)"
if [ -n "${rm_done}" ]
then
f_log "bogus adblock lists removed"
f_rmconfig "${rm_done}"
/etc/init.d/dnsmasq restart
sleep 1
check="$(pgrep -f "dnsmasq")"
if [ -n "${check}" ]
then
f_cntconfig
f_log "adblock lists with overall ${adb_count} domains loaded"
else
rc=100
f_log "dnsmasq restart failed, please check 'logread' output" "${rc}"
f_exit
fi
fi
fi
else
f_log "adblock lists with overall ${adb_count} domains are still valid, no update required"
fi
# create adblock list backups
#
if [ "${backup_ok}" = "true" ] && [ -n "${adb_revsrclist}" ]
then
backup_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_revsrclist} \) -print -exec cp -pf "{}" "${adb_dir_backup}" \;)"
rc=${?}
if [ $((rc)) -eq 0 ] && [ -n "${backup_done}" ]
then
find "${adb_dir_backup}" -maxdepth 1 -type f \( -name "${adb_dnsprefix}*" -a ! -name "${adb_dnsprefix}*.gz" \) -exec gzip -f "{}" \;
f_log "new adblock list backups generated"
elif [ $((rc)) -ne 0 ] && [ -n "${backup_done}" ]
then
f_log "error during backup of adblock lists" "${rc}"
f_exit
fi
fi
# remove temporary files and exit
#
f_exit

View File

@ -3,9 +3,9 @@
config adblock 'global'
option adb_enabled '1'
option adb_cfgver '2.1'
option adb_cfgver '2.2'
option adb_whitelist '/etc/adblock/adblock.whitelist'
option adb_whitelist_rset '\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\$1)}'
option adb_whitelist_rset '\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}'
option adb_forcedns '1'
config service 'backup'

View File

@ -59,45 +59,65 @@ start()
restart()
{
restart="true"
stop
start
}
reload()
{
restart="true"
stop
start
}
stop()
{
if [ -n "$(${adb_uci} -q get "adblock.global.adb_overall_count")" ]
cfg_check="$(${adb_uci} -q get "adblock.global.adb_overall_count")"
if [ -n "${cfg_check}" ]
then
. "/lib/functions.sh"
config_load adblock
config_foreach rm_config source
"${adb_uci}" -q delete "adblock.global.adb_dnstoggle"
"${adb_uci}" -q delete "adblock.global.adb_overall_count"
"${adb_uci}" -q commit "adblock"
"${adb_uci}" -q delete "adblock.global.adb_percentage"
"${adb_uci}" -q delete "adblock.global.adb_lastrun"
if [ -z "${restart}" ]
then
"${adb_uci}" -q commit "adblock"
fi
fi
fw_check="$(iptables -w -t nat -nL | grep -F "adb-")"
if [ -n "${fw_check}" ] && [ -z "${restart}" ]
then
iptables-save -t nat | grep -Fv -- "adb-" | iptables-restore
iptables-save -t filter | grep -Fv -- "adb-" | iptables-restore
if [ -n "$(lsmod | grep -F "ip6table_nat")" ]
then
ip6tables-save -t nat | grep -Fv -- "adb-" | ip6tables-restore
ip6tables-save -t filter | grep -Fv -- "adb-" | ip6tables-restore
fi
fi
if [ -d "${adb_dnshidedir}" ]
then
find "${adb_dnshidedir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -exec mv -f "{}" "${adb_dnsdir}" \;
fi
rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -print -exec rm -f "{}" \;)"
rc=${?}
if [ $((rc)) -eq 0 ] && [ -n "${rm_done}" ]
dns_check="$(find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -print -exec rm -f "{}" \;)"
if [ -n "${dns_check}" ]
then
rm -rf "${adb_dnshidedir}"
/etc/init.d/dnsmasq restart
/etc/init.d/firewall restart
fi
uhttpd_pid="$(pgrep -f "uhttpd -h /www/adblock")"
if [ -n "${uhttpd_pid}" ]
www_check="$(pgrep -f "uhttpd -h /www/adblock")"
if [ -n "${www_check}" ]
then
kill -9 "${uhttpd_pid}"
for pid in ${www_check}
do
kill -9 "${pid}"
done
fi
if [ -n "${rm_done}" ] || [ -n "${uhttpd_pid}" ]
if [ -n "${cfg_check}" ] || [ -n "${fw_check}" ] || [ -n "${dns_check}" ] || [ -n "${www_check}" ]
then
"${adb_log}" ${log_parm} -t "adblock[${adb_pid}] info " "all adblock related services stopped" 2>&1
fi