Spam filtering: MIT study says you can’t do it

Topic

New Reply

MIT study shows you cannot filter spam in the net/cloud or anywhere before the end user.

If you do you risk throwing away good messages.

My friend lost out on a job offer when his ISP started ‘helping’ by filtering messages without telling him and they tossed a critical email with a job offer.

The only proper way to filter spam is at the end user.

and black lists of isps or even domains often harm many innocent users because one hacker had a bot on a pc there that spewed out spam. worse than killing flies with atomic bombs.

white/black lists work best. but serious police action against the top spammers would help too.
yes you do have to check the spam folder daily to see if something good needs to be moved to the white list but that is not all that hard to do .

with any of the ‘smart’ filters you have a lot of errors and some spam slips through while it tosses out good emails. using them is not recommended if you are doing anything serious on the net. for the casual housewife forwarding stupid jokes to a bazillion people then it might be okay.

Reply | Quote

Replies

Most ISPs allow you to look at the spam folder via web access. If you periodically look at it you should be able to whitelist false positives. I will agree this is more inconvenient than turning off ISP filtering and handling it in your email client for pop email, but it is possible to properly manage it. For Imap accounts, you see the spam folder, so if you check it periodically, its easy to whitelist false positives.

Jerry

Reply | Quote

The problem with spam filtering – MIT study says you cannot do it ..

MIT study shows you cannot filter spam in the net/cloud or anywhere before the end user.

Got a link? (or did RG delete it?)

MIT use plenty of spam filtering: http://ist.mit.edu/spam

Bruce

Reply | Quote

When I was looking for a job, this almost happened to me. Fortunately I was viewing my spam folder periodically.

Reply | Quote

only works if they forward ALL the emails
then you can filter it yourself
and check the spam folder

in this case the isp deleted what they believe is spam
even though it could be valuable content to you

Reply | Quote

Who picks the threads to feature in Kathleen’s list in the newsletter? Rubbish like this front and center does WindowsSecret’s credibility no favors.

Lugh.
~
Alienware Aurora R6; Win10 Home x64 1803; Office 365 x32
i7-7700; GeForce GTX 1060; 16GB DDR4 2400; 1TB SSD, 256GB SSD, 4TB HD

Reply | Quote

@ mike feury

…

this is NOT rubbish

…

this is valuable info about what can happen to your emails that you want
and why the isps ASSuming they can filter it is incorrect

the study was from MIT (i think security oriented dept)
and is separate from whatever MIT IT folks actually do
as well as any conferences

EDITED for content

Reply | Quote

Try the “MIT Guide to Lock Picking”. If you can’t stop spam, maybe just pick it.

Reply | Quote

With all due respect to those smarter than me, why is the thread “rubbish”?

Dick

Reply | Quote

I also am not sure how this relates to the original OP’s post. Perhaps I’m missing something here. I do not see any mention of Kathleen’s list in the original post, and also do not believe this is “rubbish”. After all one person’s “rubbish” is another person’s “treasure”!

Reply | Quote

Kathleen listed this thread as “most interesting”. But we don’t know where this MIT study is, how old it is, or what it really says. (Or why the first post was edited by a moderator.)

The last MIT Spam Conference seems to have been more than three years ago: MIT Spam Conference 2010. But MIT recommend server-based filtering for their own users.

So “using them is not recommended” is only interesting because it’s not supported by anything.

Bruce

Reply | Quote

the study is very old

but the FACTS do NOT CHANGE

you can NOT accurately filter spam inside the net
it can only be done correctly at the end points

if all you do is forward jokes and phishing attempts
then you dont care

but if you want ALL your actual content delivered
then you cannot allow the isp to throw any of your emails away

Kathleen listed this thread as “most interesting”. But we don’t know where this MIT study is, how old it is, or what it really says. (Or why the first post was edited by a moderator.)

The last MIT Spam Conference seems to have been more than three years ago: MIT Spam Conference 2010. But MIT recommend server-based filtering for their own users.

So “using them is not recommended” is only interesting because it’s not supported by anything.

Bruce

Reply | Quote

the study is very old

Could you give us some hint of a reference to it? Like which decade? Because MIT do lots of filtering on their own email servers now.

Bruce

Reply | Quote

i am guessing that it is about ten years old
but it could be 15. just going by memory on this item.

what MIT IT weenies do does not reflect what the security experts in the course 6 research know

I had all my personal info hacked at GMU. they had the HR department !!! do a new database design. no surprise it ran late and had problems. In the meantime the old data had no firewall and no passwords. The security honcho said they cant make any professors follow security procedures so they dont try to use them.

what a uni actually does has no relation to real security or what some professors know

as noted earlier i lost email from my isp when they decided to filter my email on their own without any notice

i have to keep telling them to turn off the filters and let me do it

my old office mate from the uni lost a job offer when his isp did the same thing
no warnings
they just do it

we have to discover we are losing emails and then track the isp or relay down and make them stop deleting what they think is spam but we think is content

Reply | Quote

what MIT IT weenies do does not reflect what the security experts in the course 6 research know

what a uni actually does has no relation to real security or what some professors know

My take from that is that spam filtering may not work perfectly in theory, but in practice it has to be used.

Bruce

Reply | Quote

my experience says that it only needs to be used by the end points
and that doing it in the middle causes unnecessary problems

isp and IT weenies will justify imposing externalities on users
for their convenience but that is not necessary, desirable, nor proper

at the end points spam filtering works absolutely perfectly
in the middle of the net spam filtering forks the users by throwing away good content
sometimes absolutely vital content
as well as causing other problems to mere users
so that the IT weenies dont have to do more work

My take from that is that spam filtering may not work perfectly in theory, but in practice it has to be used.

Bruce

Reply | Quote

my experience says that it only needs to be used by the end points
and that doing it in the middle causes unnecessary problems

Spam filtering can still be under the control of the end user, even when it takes place on a remote server, as it is at MIT.

isp and IT weenies will justify imposing externalities on users
for their convenience but that is not necessary, desirable, nor proper

Wherever I see spam filtering, it’s always for the convenience of users, never IT staff.

at the end points spam filtering works absolutely perfectly
in the middle of the net spam filtering forks the users by throwing away good content
sometimes absolutely vital content
as well as causing other problems to mere users
so that the IT weenies dont have to do more work

Where I work, spam filtering causes a lot more work for IT staff than not filtering would do.

Bruce

Reply | Quote

at the end points spam filtering works absolutely perfectly

That’s wonderful, I haven’t succeeded in finding a perfect end-point spam filter in 15 years searching. Please share this solution, and any tips/links to making it perfect.

Where I work, spam filtering causes a lot more work for IT staff than not filtering would do.

That’s been my experience too, and also the extra load on the servers doing the anti-spam.

There are certain anti-spam jobs which can (always?) be done by the ISP or remote server better than the end-point, unless something changed in the last few years. Issues around the technical content of email transmission events and headers.

Lugh.
~
Alienware Aurora R6; Win10 Home x64 1803; Office 365 x32
i7-7700; GeForce GTX 1060; 16GB DDR4 2400; 1TB SSD, 256GB SSD, 4TB HD

Reply | Quote

you have to be funnin me

white/black/gray lists
works fine
takes one time to learn where the next email should go

you can never predict what new spam or good content may appear
so you do have to process the gray list items once

but this is 100% perfect with no good emails lost and no spam slipping through again

That’s wonderful, I haven’t succeeded in finding a perfect end-point spam filter in 15 years searching. Please share this solution, and any tips/links to making it perfect.

That’s been my experience too, and also the extra load on the servers doing the anti-spam.

There are certain anti-spam jobs which can (always?) be done by the ISP or remote server better than the end-point, unless something changed in the last few years. Issues around the technical content of email transmission events and headers.

Reply | Quote

name one thing that can be done better at teh isp than the end user

you have to assume that you know what the end user wants
or you blindly use some black list from a bogus site that claims to know who is sending spam
they routinely impose big errors on valid messages

That’s wonderful, I haven’t succeeded in finding a perfect end-point spam filter in 15 years searching. Please share this solution, and any tips/links to making it perfect.

That’s been my experience too, and also the extra load on the servers doing the anti-spam.

There are certain anti-spam jobs which can (always?) be done by the ISP or remote server better than the end-point, unless something changed in the last few years. Issues around the technical content of email transmission events and headers.

Reply | Quote

name one thing that can be done better at teh isp than the end user

Image-only email can be prevented from clogging the airwaves, ie stopped at/near source.

Lugh.
~
Alienware Aurora R6; Win10 Home x64 1803; Office 365 x32
i7-7700; GeForce GTX 1060; 16GB DDR4 2400; 1TB SSD, 256GB SSD, 4TB HD

Reply | Quote

irrelevant

if the user wants THAT email they you forked them for your convenience

only the end point can know for sure if it is content or spam

Image-only email can be prevented from clogging the airwaves, ie stopped at/near source.

Reply | Quote

i wish you worked at my isp

they keep turning it on
i have to keep telling them to stop it

most isps do it for their convenience to lower the total traffic

my friend lost a vital job offer when they bleepcanned that email
their spam his vital content. they got epsilon value. he lost big time.

Spam filtering can still be under the control of the end user, even when it takes place on a remote server, as it is at MIT.

Wherever I see spam filtering, it’s always for the convenience of users, never IT staff.

Where I work, spam filtering causes a lot more work for IT staff than not filtering would do.

Bruce

Reply | Quote

I get about 300 to 500 spam email messages every day. I was using spam protection in an attempt to better handle it, but I missed important messages (cost me money). I kept looking for better solutions. I now only filter at the end user and it’s based on my own white list/black list. It’s still letting far too much junk through, but it makes it a little more manageable. I think it’s cut down by about 2/3 the amount of junk email. I’ve personally just gotten much, much better and quickly organizing my email in preview and deleting the obvious junk. My biggest gripe is messages being sent without any subject or when the subject is written like spam, which some people do without even realizing. I’ve deleted some of those by accident, too. One last thing, ALL my email is in a master archive, so when I do accidentally delete something it’s not gone forever. From there I clean in bulk. Yah

Reply | Quote

so true
nobody in the middle can get it right
and i cannot accept type II errors

white/black/gray lists work so much better than genuine stupidity decidign for you even if they call it artificial ‘intelligence’

i cannot tolerate losing good emails
they can cost me a lot of money andor cause a lot of grief and extra work if i miss them

the good emails quickly migrate to the white list
and it is not that hard to scan the black list for an occasional new email that is wanted
better to scan a few times a day to look at a small batch
than to let them pile up and do a large number at once.

i really wish they had a standard gray list for the email i dont want in my inbox
such as newsletters
but also dont want in the trash bin to have to sort through because i knwo they are good
even if i dont want to read them asap or save them along with the real emails

I get about 300 to 500 spam email messages every day. I was using spam protection in an attempt to better handle it, but I missed important messages (cost me money). I kept looking for better solutions. I now only filter at the end user and it’s based on my own white list/black list. It’s still letting far too much junk through, but it makes it a little more manageable. I think it’s cut down by about 2/3 the amount of junk email. I’ve personally just gotten much, much better and quickly organizing my email in preview and deleting the obvious junk. My biggest gripe is messages being sent without any subject or when the subject is written like spam, which some people do without even realizing. I’ve deleted some of those by accident, too. One last thing, ALL my email is in a master archive, so when I do accidentally delete something it’s not gone forever. From there I clean in bulk. Yah

Reply | Quote

The complaint about Kathleen’s List was the fifth post in this thread, which means that opinion was based on four posts. Thanks at least in part to its being listed, there is now plenty of interest and a healthy number of interesting posts.

We are all affected by spam to a greater or lesser degree, but I receive very little real spam, which leaves me worried about how aggressive my ISP’s filters are and if there is any way of influencing that. Maybe ISPs should have a standard ‘Will that be Mild Medium Strong or Raw?’ choice for users.

The irony for me is that my paid subscription to The Newletter, yes THE Newsletter, was not received for many months this year and was only restored, for reasons unknown, with the latest edition. I did enquire at the source, with a full discussion of the technical details, and was advised that my ISP was almost certainly filtering it and to talk to them, but it somehow fell of my ‘to do’ radar, and it has just been miraculously restored.

Reply | Quote

The irony for me is that my paid subscription to The Newletter, yes THE Newsletter, was not received for many months this year and was only restored, for reasons unknown, with the latest edition. I did enquire at the source, with a full discussion of the technical details, and was advised that my ISP was almost certainly filtering it and to talk to them, but it somehow fell of my ‘to do’ radar, and it has just been miraculously restored.

I also didn’t receive it for three months and was told the same, but I get them again now. I think the sender must have crept onto a spam blacklist for a while.

Bruce

Reply | Quote

I guess I will admit that this thread started out rather “interesting”, but it turned to “rubbish” rather quickly. :o:

One can have the best of both worlds with a system like postini.com, who does filter at the server but allows me to verify everything and delete or allow as the final say.
Although they are “turning” into googleapps! Ouch1, now what?

Reply | Quote