How to download multiple files in the background with tsNet

Introduction

In order to provide a smooth user experience, it is often helpful if any files (images, videos or other data files) that need to be retrieved across the network can be downloaded by the application seamlessly in the background.

The following lesson shows how you can download a series of files in parallel and keep of track of which ones are yet to be retrieved.

Lay out the Stack

Create a new stack then drag a button and a field onto it.

Set the name of the button to "Download Files" and the name of the field to "Status".

Save this stack.

Create the Script

Edit the script of the button that you placed on the stack by right clicking on the button and selecting "Edit script".

Add the following code.

local sMaxRequests
local sActiveRequests
local sDownloadArray
local sRequestedArray
local sDataDir
local sDownloadErrors
-- The JSON data file must be in the following format:
--
-- [
--   {
--     "URL": "<url to download>",
--     "status": "",
--     "filename": "<filename to save download to>"
--   },
--   {
--     "URL": "<url to download>",
--     "status": "",
--     "filename": "<filename to save download to>"
--   }
-- ]
--
on mouseUp
   -- Make sure we are not already downloading files in the background
   if sActiveRequests is not 0 then
      answer "Downloads currently in progress."
   end if
   
   -- Set a limit for the number of requests we want to perform at a time
   put 3 into sMaxRequests
   
   -- This will be set to true if an error occurs while downloading any of the files
   put false into sDownloadErrors
   
   -- Reset the array of requested downloads
   put empty into sRequestedArray

   -- Specify the folder that downloaded files will be stored in
   put specialFolderPath("Desktop") into sDataDir
   
   -- Retrieve the JSON data file which contains the files to download and the current download status of each
   -- and convert it to an array called sDownloadArray
   put URL("file:" & sDataDir & "/URLs.json") into tJsonData
   put JSONToArray(tJsonData) into sDownloadArray
   put "Downloading..." into field "Status"
   
   -- Begin downloading
   send "downloadURLs" to me in 0 milliseconds
end mouseUp
on downloadURLs
   put the number of lines in the keys of sDownloadArray into tArrayCount
   
   -- If we are not already downloading the maximum number of simultaneous files, loop through
   -- sDownloadArray and initiate a request to an entry that is not yet "downloaded"
   if sActiveRequests < sMaxRequests then
      repeat with x = 1 to tArrayCount
         if sDownloadArray[x]["status"] is not "downloaded" and sRequestedArray[x] is not true then
            put tsNetGet(x, sDownloadArray[x]["URL"], tHeaders, "transferComplete") into tResult
            
            -- Increment the number of active requests and flag this item as having been requested
            if tResult is empty then
               add 1 to sActiveRequests
               put true into sRequestedArray[x]
            end if
         end if
         
         -- Stop downloading new files if we have reach the maximum number of simultaneous downloads
         if sActiveRequests = sMaxRequests then 
            exit repeat
         end if
      end repeat
   end if
   
   -- If we have looped through the array and no downloads are occurring, we have finished processing
   -- all the files to download, so execute the "downloadsCompleted" handler
   if sActiveRequests = 0 then
      downloadsCompleted
      exit downloadURLs
   end if
   
   -- Keep looping if we are not yet finished
   send "downloadURLs" to me in 50 milliseconds
end downloadURLs
on transferComplete pID, pResult, pBytes, pCurlCode
   -- Decrement the number of active requests as one has just finished
   subtract 1 from sActiveRequests
   
   -- Check the status codes and look for success
   if pCurlCode is 0 and pResult is 200 then
      -- If successful, retrieve the downloaded data
      put tsNetRetrData(pID, tError) into tData
      
      if tError is empty then
         -- Downloaded data has been retrieved successfully, save the data to the filesystem
         -- and update the status of the URL in sDownloadArray 
         put "downloaded" into sDownloadArray[pID]["status"]
         put tData into URL("file:" & sDataDir & slash & sDownloadArray[pID]["filename"])
         
         -- Save the array back to the filesystem so that if the application is closed prior
         -- to all files being downloaded, we can pick up where we left off
         put ArrayToJSON(sDownloadArray,,true) into URL("file:" & sDataDir & "/URLs.json")
      end if
   else
      -- A transfer must have failed, flag the failure and update the URL's status in sDownloadArray
      put true into sDownloadErrors
      put "failed" into sDownloadArray[pID]["status"]
      put ArrayToJSON(sDownloadArray,,true) into URL("file:" & sDataDir & "/URLs.json")
   end if
   
   -- Always call tsNetCloseConn to release any memory associated with the transfer
   tsNetCloseConn pID
end transferComplete
on downloadsCompleted
   -- Check for any download errors during processing, and notify the user accordingly
   if sDownloadErrors is true then
      put "All downloads attempted, but at least one was unsuccessful. Click the button to retry any failed downloads."  into field "Status"
   else
      put "All downloads completed successfully." into field "Status"
   end if
end downloadsCompleted
Click to copy

Create a file

The button script above accesses a text file to retrieve and maintain the list of URLs that must be downloaded.

Open up your favourite text editor program and copy the following text into your file.  Name this file "URLs.json" and save it to your desktop.

[
  {
    "URL": "https://www.livecode.com",
    "filename": "livecode.txt",
    "status": ""
  },
  {
    "URL": "https://www.google.com",
    "filename": "google.txt",
    "status": ""
  },
  {
    "URL": "https://www.microsoft.com",
    "filename": "microsoft.txt",
    "status": ""
  },
  {
    "URL": "https://www.apple.com",
    "filename": "apple.txt",
    "status": ""
  }
]
Click to copy

Test

Now you are ready to test your coding and see the results.

Switch to Run mode and click the button.

More information

This example uses a text file in JSON format to maintain the list of URLs to download and their current download status.  However, this information could be stored in any number of formats.  It could also be retrieved from a remote URL or stored in a database.

14 Comments

Mike Felker

I am having a problem.

I need to achieve this without server side scripting. I need to download multiple files all within the same livecode program. How would I do this?

Right now, if I try to download files from a server (good internet connection and working server assumed) using SFTP with a repeat / end repeat loop, it works sporadically. Unless I issue a answer box in the loop, it may or may not reliably download each small file (under 5k each). If I issue a wait timer of some kind, it does not work. However, if I issue a answer box (like answer "The file has been downloaded" with "OK" - that makes it work). Is there some command I am missing that allows files to be downloaded in the background without user interaction to be successfully downloaded every single time?

Thanks.

Mike

Matthias Rebbe

You are using tsNET, right. I´ve ran into a similar problem and solved it by setting
tSettings["no_reuse"] to TRUE, which forces tsNet to disconnect the connection to the server after each transfer.

The dictionary says
"no_reuse": Set to true to specify that the connection to the server should be disconnected and not left open for any future connections.

At least this helped me to solve it, but you could give it a try.

Mike Felker

I tried this and it did not work. I added:

put true into tSettings["use_ssl"]
put true into tSettings["no_reuse"]
put true into tSettings["save_sent_headers"]

but it still cycles through the loop too fast and does not download anything.

Mike

Mike Felker

Wait timers don't work either. I tried:

wait 5 seconds

in the loop and it does wait 5 seconds, but does not initiate a download.

What is weird is that if I replace that line with:

answer "Whatever text I want" with "OK"

then it works fine.

Mike

Matthias Rebbe

Hm, so let´s hope that Charles Warwick as the developer of tsNET or Eleanor chime in. Maybe they have an idea. The people at Livecode Ltd. are back to work next week. Could you post the script, of course without any real login or url data, one could test? I would like to try with my server.

Mike Felker

I am getting a "ID already in use error". How do we clear the ID before each new download in the loop?

Mike

Mike Felker

I can only post a bit of it - we have patents pending.
Server variables and username/password variables are ommitted.

Assume there are 10 files:

[code]

-- Download from server
-- Store the SFTP server details in some variables
put "sftp://myserver" into tSFTPServer
put "/path/to/file" into tLocalRegFile
put "/path/to/user/file" into tUserFile
put true into tSettings["use_ssl"]
put true into tSettings["no_reuse"]
put true into tSettings["save_sent_headers"]

repeat with x = 10 down to 1
## If we make the below line active, then it works as long as you wait long enough for the file to download
--answer "Downloading Block " & x

## Create files to download
put tUserFile & "_" & x & ".bl" into tRemoteDownload
put tLocalRegFile & "_" & x & ".bl" into tLocalDownload

put tsNetGetFile("mySFTPdownload", tLocalDownload, tSFTPServer & tRemoteDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult

if tResult is not empty then
-- Inform the user of the error
put "There has been an error. Error details:" && tResult into field "fldStatus"
## We could warn with a answer box
--answer "There has been an error"
else
## Download was a success
put "Success" into field "fldStatus"
end if
end repeat
end opencard

## TSNet required code

on tsNetGetFileFinished pID, pResult, pBytes, pCurlCode
put "tsNetGetFile request has completed, checking result - code: " &pCurlCode

-- If pCurlCode is anything other than 0, the request has not been successful.
-- pResult will also be 0 for successful SFTP transfers

-- If either of these are not the case, we will inform the user
if pCurlCode is not 0 or pResult is not 0 then
answer "Could not retrieve data from site"

else
-- The file has been downloaded and stored in the location indicated by the tLocalPath variable
put "File has downloaded successfully" into field "fldStatus"
end if

-- Tidy up the resources used by the tsNetGetFile call:
tsNetCloseConn pID
end tsNetGetFileFinished
[/code]

That's what I can show. Thanks for any help.

Mike

Matthias Rebbe

Which tsNet command are you using? The connectionID, which is needed for non-sync tsNet commands, is user defined. So you have to define an ID for each download request. How does your script looks like?

Mike Felker

put tsNetGetFile("mySFTPdownload", tLocalBlockToDownload, tSFTPServer & tRemoteBlockToDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult

Mike Felker

How would I assign a ID in the loop?

Mike

Mike Felker

put tsNetGetFile("mySFTPdownload", tLocalBlockToDownload, tSFTPServer & tRemoteBlockToDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult

How would I set up a new ID within the loop?

Mike

Matthias Rebbe

Currently your connectionID is always mySFTPdownload

You could change
put tsNetGetFile("mySFTPdownload", tLocalBlockToDownload, tSFTPServer & tRemoteBlockToDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult

to

put tsNetGetFile(x, tLocalBlockToDownload, tSFTPServer & tRemoteBlockToDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult

so that the connectionID now contains the value of X, which is the loop counter or however this might be called.

Mike Felker

You are AWESOME Matthias! All fixed. Works perfectly with a 2 second wait command.

Thank you!

Mike

Matthias Rebbe

Great to hear that it works now.

Add your comment

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.