automatically downloading story stats

[joy_of_cooking]

Hey everyone, I just figured out how to download my story stats without manually clicking the link:

curl https://literotica.com/api/3/submissions/my/stories/published.csv --cookie 'auth_token=...'

The "..." is specific to your login session. Once you're logged in to the site in a web browser, you can get your auth token by using the network inspector to record the outgoing request when you click the link in your browser. I don't yet know how long-lived this value is.

 

[onehitwanda]

 

[intim8]

I just figured out how to download my story stats without manually clicking the link:

Anybody know if doing this kind of thing is a problem with the TOS?

 

[joy_of_cooking]

Anybody know if doing this kind of thing is a problem with the TOS?

I don't know, but I doubt the site's owners would care as long as you throttle it to something reasonable for interactive use. Hourly would probably be safe, considering how often, uh, some friends of mine refresh their control panels shortly after publishing a story.

 

[alohadave]

The "..." is specific to your login session. Once you're logged in to the site in a web browser, you can get your auth token by using the network inspector to record the outgoing request when you click the link in your browser. I don't yet know how long-lived this value is.

What specifically are we looking for?

Edit: I found it. The ' around the auth_token broke it. Taking it out makes it work.

 

[Duleigh]

{
"success": false,
"message": "Check query parameters. The selected status is invalid."
}

Your script hates me

 

[Duleigh]

All you really need is this:

https://literotica.com/api/3/submissions/my/stories/published.csv

Click that link and away you go!

 

[oneagainst]

Would suspect not... you have to log in first, so it's not like scraping the entire site. It'd be good if we could have a persistent auth token we could use in the same place as the session to set up a script for this, which could then harvest the daily CSV download into a data lake from which we can get realtime stats on story performance.

Hmm, if enough of us did that, we could feed it into an AI model that could tell us which story types yield the most views, then potentially train a large language model on the story corpus to autogenerate those

 

[intim8]

All you really need is this:

If you want to do it manually. Scripting it is a different story.

 

[Bramblethorn]

Would suspect not... you have to log in first, so it's not like scraping the entire site. It'd be good if we could have a persistent auth token we could use in the same place as the session to set up a script for this, which could then harvest the daily CSV download into a data lake from which we can get realtime stats on story performance.

Hmm, if enough of us did that, we could feed it into an AI model that could tell us which story types yield the most views, then potentially train a large language model on the story corpus to autogenerate those

But let's not

 

[intim8]

It'd be good if we could have a persistent auth token we could use in the same place as the session to set up a script for this, which could then harvest the daily CSV download into a data lake from which we can get realtime stats on story performance.

It'd be better if Lit had a formal API. One that could do more than just this.

 

[joy_of_cooking]

It'd be good if we could have a persistent auth token we could use in the same place as the session

I'm completely speculating but I suspect the auth token doesn't expire. It would be the simplest explanation for the fact that I never have to log back in.

 

[MillieDynamite]

Oh, my God, y'all need to get a life outside of Lit, gang!

 

[NotWise]

Anybody know if doing this kind of thing is a problem with the TOS?

I don't recall the TOS saying anything about it, but it's been a long time since I read it. Manu created a facility to let you download your story stats as a CSV file (download symbol on your works page, right end of the search bar), so I doubt it's against the TOS.

I've never tested the longevity of the authority very precisely, but my recollection from various tests and experiments was that it had to be refreshed every 1 1/2 to 2 hours.

 

[NotWise]

I'm completely speculating but I suspect the auth token doesn't expire. It would be the simplest explanation for the fact that I never have to log back in.

The token is probably refreshed automatically in the background.

 

[dmallord]

Hey everyone, I just figured out how to download my story stats without manually clicking the link:

curl https://literotica.com/api/3/submissions/my/stories/published.csv --cookie 'auth_token=...'

The "..." is specific to your login session. Once you're logged in to the site in a web browser, you can get your auth token by using the network inspector to record the outgoing request when you click the link in your browser. I don't yet know how long-lived this value is.

I beg your pardon, but isn't it just as easy to click the link? Where do you keep all that code and extra stuff like auth token, whatever that is?

 

[intim8]

I beg your pardon, but isn't it just as easy to click the link? Where do you keep all that code and extra stuff like auth token, whatever that is?

Not if you want to do it regularly and have it pulled into something that will track it.

 

[shakna]

The auth token lasts either:

* Next time you login, it gets revoked. (A new login session, that is.)

* 30 days.

 

[joy_of_cooking]

The auth token lasts either:

* Next time you login, it gets revoked. (A new login session, that is.)

* 30 days.

How do you know?

 

[NotWise]

The auth token lasts either:

* Next time you login, it gets revoked. (A new login session, that is.)

* 30 days.

I have an app that monitors my stats. The interval between checks is usually four hours, but less when there's enough traffic. At the normal 4-hour interval the auth token has to be renewed every time it checks. It doesn't have to be renewed every time if the interval is less than about 1 1/2 hours.

I made a quick check on line, and "they" say that the interval varies.

At any rate, renewing the auth token is quick and easy. According to my logs, the whole cycle of starting a download, getting the "expired" notice, renewing the token, and restarting a download is pretty quick. This is a log excerpt with the most recent entry at the top:

2024-03-15 03:58:08 - Downloading
2024-03-15 03:58:07 - Authorizing
2024-03-15 03:58:07 - JWT has expired. Renew
2024-03-15 03:58:07 - Downloading

 

[joy_of_cooking]

Tell me more about this app @NotWise

 

[oneagainst]

Oh, my God, y'all need to get a life outside of Lit, gang!

A stranger... from the outside....

 

[shakna]

How do you know?

Couple of years running a script that collates my stats for me.

 

[joy_of_cooking]

Couple of years running a script that collates my stats for me.

Whoa, tell me more!

 

[NotWise]

Tell me more about this app @NotWise

It's been running on my desktop for a couple years now, and it looks like this:



The picture is Louise Brooks (one of my favorite photo subjects), but there are about a dozen other photos in a library that I can flip through to get a different view. Mostly those photos are there to fill the right side of the window. I haven't bothered to label the columns, but they're title, logged time, rating, views and votes. Clicking "Recent events" gives you the event log. Mousing over the hourglass icon gives you the time to the next update. The buttons are almost self-explanatory. You can reset the monitoring interval, pause the process, check now or exit.

The whole thing started years ago with spreadsheets and a program that tried to back-calculate missing votes. I got tired of that, so I decided to automate the process as much as I could. I started with a shell script that downloaded the csv file at predetermined intervals. That original script used "curl" to take care of all the http details, and the more developed app still does.

From there, I developed a ********** code that did the whole thing on a web page. That had too many requirements, didn't give me enough control, and couldn't be expanded to do some things I wanted, so I rewrote it in C with GTK3 for the GUI. As the last step, I expanded it and updated it to use GTK4. The style settings are mostly defaults for GTK4 because I don't grok their style sheets.

For most people, Python might be a better choice than C. There are other choices.

The monitor front-ends a lot of tasks that run in the background. The downloaded stats are stored in an SQLite database, and missing votes are back-calculated and/or estimated with a more-developed version of the code that I used with the spreadsheets. The story entries on the scrolling list link to a pop-up menu that lets me open the database table for the story or produce graphs of the story's results: views vs elapsed time, votes vs elapsed time, votes vs views.

The graphs contain a line that fits a model to the data and tries to characterize the viewing rate and population sizes for different viewer populations ("new list" readers, "contest readers", "at large" readers). I use GLE to produce the graphics, and ImageMagik to display them. The curve-fitting program is complicated, but that's all imbedded, and I don't have to think about it.

I haven't tried to integrate it with Ubuntu's desktop system, so it's all self-contained. It and all of its moving parts reside in one folder.

The monitor downloads the csv file every four hours, unless any one story on the list gets two or more votes in the interval, then it cuts the interval in half, down to a minimum of 8 minutes. If no story gets more than one vote in an interval, than it doubles the interval back to four hours.

It's behavior with new stories has been problematic, but It's been running without significant revision for almost two years, with downtime for power outages and system updates. It's stable. I figure that producing the CSV file is a lot smaller load on Lit than producing the "works" page, and checking at intervals results in a lot less traffic than updating the works page every time I think about it, which was sometimes very frequently.

edit: Apparently the forum doesn't want you to name the world's most commonly-used programming language--usually for browser-based applications.