Piano Society
Free Classical Keyboard Recordings
It is currently Sat Oct 25, 2014 5:15 am

All times are UTC - 1 hour




Post new topic Reply to topic  [ 47 posts ]  Go to page Previous  1, 2, 3, 4  Next
Author Message
 Post subject: Re: problems with statistics
PostPosted: Wed Jan 05, 2011 4:36 pm 
Offline
Site Admin

Joined: Mon Jun 12, 2006 11:45 am
Posts: 9575
Location: Netherlands
You mean you fixed the stats ??? That was quick.

_________________
Nothing is always absolutely so -- Sturgeon's law
Chris Breemer


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Wed Jan 05, 2011 5:04 pm 
Offline
Site Admin

Joined: Wed Jun 14, 2006 12:38 pm
Posts: 8514
:roll: :lol:

_________________
"Simplicity is the highest goal, achievable when you have overcome all difficulties." ~ Frederic Chopin

my videos: http://www.youtube.com/user/monicapiano
my personal website: http://www.monicaalianello.com


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Wed Jan 05, 2011 5:22 pm 
Offline

Joined: Mon Nov 29, 2010 7:28 am
Posts: 1250
Location: Springfield, Missouri, USA
Impressive! Speaking of stats, judging by the posts and replies (and the number of votes on my poll for whose music to only live with elsewhere), as a newcomer I have found the active PS Community to be much smaller than expected, but thouroughly enjoyable nonetheless and I'm happy that I joined :) .

_________________
Eddy M. del Rio, MD
"A smattering will not do. They must know all the keys, major and minor, and they must literally 'know them backwards.'" - Josef Lhevinne


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Thu Jan 06, 2011 9:16 am 
Offline
Site Admin

Joined: Mon Jun 12, 2006 11:45 am
Posts: 9575
Location: Netherlands
musical-md wrote:
Impressive! Speaking of stats, judging by the posts and replies (and the number of votes on my poll for whose music to only live with elsewhere), as a newcomer I have found the active PS Community to be much smaller than expected, but thouroughly enjoyable nonetheless and I'm happy that I joined :) .

Good to hear Eddy, and do stick around. Too many people have left over the years - this used to be a much livelier place with more discussion and fun. On the positive side, the quality of both postings and submissions has hugely improved since then.

_________________
Nothing is always absolutely so -- Sturgeon's law
Chris Breemer


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Thu Jan 06, 2011 10:20 am 
Offline
Site Admin

Joined: Mon Jun 12, 2006 11:45 am
Posts: 9575
Location: Netherlands
Ok, ok.... I'll work on fixing the stats. I found that in february last year I had already laid the groundwork, i.e. managing and filtering the server access log. I had forgotten all about that :D

This might explain a hiccup that some people experience now and then. Every midnight, the webserver is briefly brought down to be able squirrel away a copy of the access log and start with a clean slate (the access log becomes too huge to handle otherwise). I don't believe this takes more than a couple of seconds, or maybe minutes at worst, but it is a little window of unavailability all the same.

_________________
Nothing is always absolutely so -- Sturgeon's law
Chris Breemer


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Fri Jan 07, 2011 8:59 am 
Offline

Joined: Mon Dec 27, 2010 1:11 am
Posts: 243
Location: Adelaide, Australia
techneut wrote:
That someone would have to dig around in my C source code.


I'd be willing to look at the source code and see whether I'm able to help. (The answer might well be "no", since it's about 10 years since I last programmed in C. But I'm kind of curious as to how much I can remember, and it costs nothing to look.) Message me if you want to follow up on this; it's probably not a good idea to post the source code in a public forum.

_________________
Alexander Hanysz, http://hanysz.net


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Fri Jan 07, 2011 11:50 am 
Offline
Site Admin

Joined: Mon Jun 12, 2006 11:45 am
Posts: 9575
Location: Netherlands
hanysz wrote:
I'd be willing to look at the source code and see whether I'm able to help. (The answer might well be "no", since it's about 10 years since I last programmed in C. But I'm kind of curious as to how much I can remember, and it costs nothing to look.) Message me if you want to follow up on this; it's probably not a good idea to post the source code in a public forum.

Thanks for the offer, but even if C coding was your daily bread today this would not help so much. It's not that the C code is bad or buggy, it's just that the external conditions have changed. We now have a different way of handling the access logs, and the program needs to be adapted to that. As well as simplified so that I can still understand my own logic after a couple of months :D

But if you're just curious I can mail you the source code.

_________________
Nothing is always absolutely so -- Sturgeon's law
Chris Breemer


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Sun Jan 09, 2011 12:21 am 
Offline

Joined: Mon Dec 27, 2010 1:11 am
Posts: 243
Location: Adelaide, Australia
Indeed I am curious :-)

_________________
Alexander Hanysz, http://hanysz.net


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Sun Jan 09, 2011 1:15 pm 
Offline
Site Admin

Joined: Mon Jun 12, 2006 11:45 am
Posts: 9575
Location: Netherlands
I am now under steam with rewriting the code as needed. It's starting to do something again. I'll send you the code when it's ready.

_________________
Nothing is always absolutely so -- Sturgeon's law
Chris Breemer


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Wed Jan 12, 2011 10:38 pm 
Offline
Site Admin

Joined: Mon Jun 12, 2006 11:45 am
Posts: 9575
Location: Netherlands
The stats program is shaping up again, I think I'm almost ready to regenerate the full stats from february 2010 until now. I more or less needed to overhaul the entire C code. The cumulative stats has to process ~7 million downloads and sort them in various ways. This was eating up memory like crazy and I had to re-think the way this enormous wad of data was stored in memory. Had some terrible time today with untraceable heap corruptions but luckily they disappeared after I simplified a certain part of the code :D
All in all quite a project but the end seems near now. Would be great to get back to the piano again :roll:

_________________
Nothing is always absolutely so -- Sturgeon's law
Chris Breemer


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Wed Jan 12, 2011 11:17 pm 
Offline

Joined: Sun Nov 30, 2008 1:11 am
Posts: 489
Location: Lyon, France
Good to read that, Chris. Don't fall in the last turn !

_________________
François
"Je préfère ce qui me touche que ce qui me surprend"
François Couperin (1668-1733)


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Thu Jan 13, 2011 7:07 am 
Offline

Joined: Mon Dec 27, 2010 1:11 am
Posts: 243
Location: Adelaide, Australia
Thanks Chris for doing this! I hope it hasn't been too unpleasant a task.

_________________
Alexander Hanysz, http://hanysz.net


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Sun Jan 16, 2011 9:54 am 
Offline
Site Admin

Joined: Mon Jun 12, 2006 11:45 am
Posts: 9575
Location: Netherlands
At long last, the download statistics have risen from the ashes. It took me a long time to get all the debris cleaned up and start with a clean slate. We now have statistics dating back to feb. 2010. I have once more added the 'Most active downloaders' column, from which we can see (by clicking the links) that many of the power downloaders are located in China.

A few words about interpreting these numbers.

First of all I make no guarantee that these numbers are 100% accurate. The amount of data trawled is enormous and I have to make educated guesses on what to process and what to ignore. These numbers should be seen as more than an indication.

Next, I strongly suspect that there are many sites who regularly scrape ALL recordings from our site. Because of that, it stands to reason that whoever has the most recordings is always the most downloaded artist. In fact I think these numbers are directly related to the number of tracks on the site. For example my download figure number is roughly 3 times higher than that of Monica, and I have roughly 3 times as many recordings on the site. So really these figures do not prove a lot other than the law of large numbers.

Also, I suspect there are certain sites that have a 'recording of the day/week/month' and play it over and over, directly from the site, thus skyrocketing the numbers for that recording and artist. For example during may, june, and july 2010 someone was really fond of one little item from my Mikrokosmos cycle. In december, an old recording of Chopin's op.66 tops the charts. There seems to be no rule to it, except it seems to happen more for people with lots of recordings. It would be interesting to correlate this to the power downloaders to see who's doing this. An exercise for another day.

Currently, the nightly (midnight European time) process of generating the full stats takes about 15 minutes, and a lot of memory, during which time I could imagine some disruption of services. This will of course slowly increase over time. We'll have to see if and when that becomes a problem. I really need to think about a way to update the statistics each day instead of completely regenerate them. That's for another day also.

If anybody finds any mistakes in the stats (missing/wrong links, duplicates, stuff that should not be there, etc.....) please let me know. I noticed in one page that Bach is listed twice, but with all the Bach transcriptions that doesn't surprise me and I can't be bothered to fix it. Things like double names make maintaining the stats a PITA.

Hm, I see that now the New Recordings list is a bit lame... the artist name is missing. I think I know why and will fix that later today.

_________________
Nothing is always absolutely so -- Sturgeon's law
Chris Breemer


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Sun Jan 16, 2011 2:33 pm 
Offline

Joined: Sun Nov 30, 2008 1:11 am
Posts: 489
Location: Lyon, France
Hi Chris,
Thanks for the news about the progress of this big site. However, to date the stats are still not accessible, are they ?

_________________
François
"Je préfère ce qui me touche que ce qui me surprend"
François Couperin (1668-1733)


Top
 Profile  
 
 Post subject: Re: problems with statistics
PostPosted: Sun Jan 16, 2011 2:36 pm 
Offline
Site Admin

Joined: Mon Jun 12, 2006 11:45 am
Posts: 9575
Location: Netherlands
Yes they are !

_________________
Nothing is always absolutely so -- Sturgeon's law
Chris Breemer


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 47 posts ]  Go to page Previous  1, 2, 3, 4  Next

All times are UTC - 1 hour


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group