15 Commits

Author SHA1 Message Date
0f5341835f Add version check
Close #11
2025-03-05 01:06:44 +01:00
0196476370 Add built-in scheduler when running --webui
Also:
- Default to webui

Close #16
2025-03-04 20:03:34 +01:00
1f81d48aad Open browser when starting server with --webui
Close #15
2025-03-04 14:26:35 +01:00
e3ac9e9b41 Add makefile, add function for getting version 2025-03-04 14:06:07 +01:00
b7923c9091 Finish first version of WebUI
Close #10
2025-03-04 12:05:31 +01:00
dcc46f679b Add option to upload files in webui 2025-03-03 19:42:53 +01:00
a8a6292d2b Begin adding WebUI
See #10
2025-03-02 22:59:51 +01:00
5abc1fe75a Update readme 2025-02-28 03:39:29 +01:00
7cfd3bb1de First working version
- Move from separate dirs for upload to a unified one (identify
  media/metadata files by file extension)
- Prevent uploading when an import is already scheduled
- Allow setting custom, parser-specific settings
- Add CLI
- Implement WebDAV upload
- Implement checking of upload folders for uploadable contents

Close #6, close #7, close #9, close #3, close #1, close #4
2025-02-27 17:29:20 +01:00
9a5c432186 Remove and ignore binary from git 2025-02-26 13:49:04 +01:00
565b183d17 Add CLI for setting settings 2025-02-26 03:23:43 +01:00
758495e3ee Start adding cli 2025-02-26 00:21:04 +01:00
5afc375042 Set user agent in getting HTTP status code
Close #8
2025-02-25 22:14:27 +01:00
3cb49d005e Implement loading and storing config to / from file 2025-02-25 04:00:30 +01:00
d4c83e27d2 Add add config validators 2025-02-24 04:33:43 +01:00
44 changed files with 3321 additions and 99 deletions

3
.gitignore vendored
View File

@@ -1 +1,2 @@
go.work
/go.work
/bin

3
.gitmodules vendored Normal file
View File

@@ -0,0 +1,3 @@
[submodule "src/webui/translation-json"]
path = src/webui/translation-json
url = https://gitea.armuli.eu/museum-digital/translation-json.git

View File

@@ -1,16 +1,35 @@
# Uploader to automate the Upload and Import of Objects in museum-digital via WebDAV
The purpose of this app will be to allow museums to more simply automate the upload of
their collection data to museum-digital. It is thus targetted at museums using museum-digital
their collection data to museum-digital. It is thus targeted at museums using museum-digital
as a publication platform, while using another collection management system.
While collection management systems differ, an automatic import to museum-digital works in
a uniform way. Import data is uploaded via WebDAV, an import is configured using a text file. ([See here](https://de.handbook.museum-digital.info/import/importe-selbst-durchfuehren.html))
a uniform way. Import data is uploaded via WebDAV, an import is configured using a text file.
([See here](https://de.handbook.museum-digital.info/import/importe-selbst-durchfuehren.html))
This part of the automatic importing can thus be solved in a one-fits-all solution here.
Collection management system-specific export scripts are necessary to present the present
app with data to upload. The export itself is thus out of the scope of the tool described here.
## Idea / Very Short Summary / Use Cases
### Automatic Publication from Separate Collection Management System
A museum wants to use museum-digital only for publication, while they use a separate collection
management system. They want to keep their published collections up to date with their local data.
Hence, they automate an export from their local program. To upload, they move the data into a
specified folder. The present application monitors the folder and uploads the data to museum-digital.
### Automating Image Uploads
If a museum produces a lot of digital media files (e.g. lots of scans of books / manuscripts),
it may become tedious to upload them manually. This becomes an issue all the more if there are
lots of uncompressed, high resolution files (again, common with scans).
In this case, it may be useful to use this application to automatically upload the object images
for import using the `ImageByImport` parser - a.k.a. the matching of image files to objects based
on their file name according to pre-defined rules.
## Workflow
### Setup
@@ -26,8 +45,7 @@ It will hence attempt to do so when first run (or unless all relevant informatio
- Auth token for WebDAV access
- Institution ID
- Parser
- Location of metadata files to upload
- Location of images files to upload
- Location of uploadable files
- Decision, whether objects should be imported as public
#### Setup workflow
@@ -63,26 +81,19 @@ It will hence attempt to do so when first run (or unless all relevant informatio
## Commands
- [ ] Upload
- [x] Upload
- [ ] Verify ability to connect
- [ ] Set optional setting (do the settings need to be validated?)
- [ ] List config
- [x] Set optional setting (do the settings need to be validated?)
- [x] List config
- [ ] Remove optional setting (in case an erroneous setting value has been set)
- [ ] `--version` command to check currently installed version
- [ ] Version check (via Gitea release RSS feed)
## Interface
- A command line interface is sufficient for this application
## Libraries/resources to include
- `requests` / HTTP library
- JSON parser
- XML parser (for version/update check)
- WebDAV client: [Python](https://pypi.org/project/webdav4/), [Golang](https://github.com/studio-b12/gowebdav)
Either JSON or XML may also serve to store the config data.
- A command line interface is sufficient for this application. It would be nice to
eventually provide a GUI for running settings as well as to provide some GUI for
warnings to update the app.
## Additional, optional features

10
go.mod
View File

@@ -1,3 +1,13 @@
module gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader
go 1.24.0
require (
github.com/go-co-op/gocron/v2 v2.16.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/jonboulle/clockwork v0.5.0 // indirect
github.com/madflojo/tasks v1.2.1 // indirect
github.com/robfig/cron/v3 v3.0.1 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/studio-b12/gowebdav v0.10.0 // indirect
)

12
go.sum Normal file
View File

@@ -0,0 +1,12 @@
github.com/go-co-op/gocron/v2 v2.16.0 h1:uqUF6WFZ4enRU45pWFNcn1xpDLc+jBOTKhPQI16Z1xs=
github.com/go-co-op/gocron/v2 v2.16.0/go.mod h1:opexeOFy5BplhsKdA7bzY9zeYih8I8/WNJ4arTIFPVc=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/jonboulle/clockwork v0.5.0 h1:Hyh9A8u51kptdkR+cqRpT1EebBwTn1oK9YfGYbdFz6I=
github.com/jonboulle/clockwork v0.5.0/go.mod h1:3mZlmanh0g2NDKO5TWZVJAfofYk64M7XN3SzBPjZF60=
github.com/madflojo/tasks v1.2.1/go.mod h1:/WMv6u3Xb5eyy+aIM76ildaIT166GOxN/jya9oI7dyo=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/studio-b12/gowebdav v0.10.0 h1:Yewz8FFiadcGEu4hxS/AAJQlHelndqln1bns3hcJIYc=
github.com/studio-b12/gowebdav v0.10.0/go.mod h1:bHA7t77X/QFExdeAnDzK6vKM34kEZAcE1OX4MfiwjkE=

66
main.go
View File

@@ -1,38 +1,42 @@
package main
import "fmt"
import (
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configLoader"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/setup"
"slices"
"os"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/cli"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/webui"
)
// Attempts to connect to WebDAV server, verifying that the authentication data
// works and that the server is one for uploads with museum-digital.
// A museum-digital WebDAV directory can be identified by the availability of the
// directories IMPORT_IMG and IMPORT_XML.
func verifyConnection(config MDWebDavUploaderConfig) bool {
}
// Generates the contents of the file import_config.txt necessary
// to let museum-digital know about the required import configuration.
func generateImportConfigFile() string {
}
// Uploads metadata files to the configured instance of
// museum-digital / musdb.
func uploadMetadataFiles() {
}
// Uploads media files to the configured instance of
// museum-digital / musdb.
func uploadMediaFiles() {
}
// Negotiates commands based on command line options.
func main() {
fmt.Println("hi")
if slices.Contains(os.Args, "--help") {
} else if slices.Contains(os.Args, "--version") {
cli.ShowVersion();
} else if slices.Contains(os.Args, "--check-update") {
cli.CheckVersionIsUpToDate();
} else if slices.Contains(os.Args, "--show-config") {
cli.ShowCurrentConfig()
} else if slices.Contains(os.Args, "--run-manual-setup") {
cli.RunManualSetup()
} else if slices.Contains(os.Args, "--set-additional-setting") {
cli.SetAdditionalSetting()
} else if slices.Contains(os.Args, "--upload") {
cli.Upload()
} else if slices.Contains(os.Args, "--webdav-remote-list-toplevel") {
cli.ListRemoteToplevel()
} else if slices.Contains(os.Args, "--webdav-remote-list-metadata-dir") {
cli.ListRemoteMetadataDir()
} else if slices.Contains(os.Args, "--webdav-remote-list-media-dir") {
cli.ListRemoteMediaDir()
} else if slices.Contains(os.Args, "--local-list-metadata") {
cli.ListLocalMetadata()
} else if slices.Contains(os.Args, "--local-list-media") {
cli.ListLocalMedia()
} else if slices.Contains(os.Args, "--webui") {
webui.Run()
} else {
webui.Run()
}
}

22
makefile Normal file
View File

@@ -0,0 +1,22 @@
VERSION := $(shell go run main.go --version)
hello:
echo WebDAV uploader for museum-digital
echo Version is $(VERSION)
build:
go build -o bin/museum-digital-webdav-uploader main.go
run:
go run main.go
compile:
echo "Compiling for every OS and Platform"
GOOS=linux GOARCH=amd64 go build -o bin/museum-digital-webdav-uploader-$(VERSION)-linux-amd64 main.go
GOOS=windows GOARCH=amd64 go build -o bin/museum-digital-webdav-uploader-$(VERSION)-windows-amd64.exe main.go
GOOS=darwin GOARCH=amd64 go build -o bin/museum-digital-webdav-uploader-$(VERSION)-macos-amd64 main.go
GOOS=linux GOARCH=arm64 go build -o bin/museum-digital-webdav-uploader-$(VERSION)-linux-arm64 main.go
GOOS=windows GOARCH=arm64 go build -o bin/museum-digital-webdav-uploader-$(VERSION)-windows-arm64.exe main.go
GOOS=darwin GOARCH=arm64 go build -o bin/museum-digital-webdav-uploader-$(VERSION)-macos-arm64 main.go
all: hello build

53
src/cli/Upload.go Normal file
View File

@@ -0,0 +1,53 @@
package cli
import (
"fmt"
"os"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/uploadsrcdir"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/webdavupload"
)
// Integration: Uploads, if there is uploadable data and no import currently
// scheduled.
func Upload() {
config := loadConfigFromFile()
uploadableMetadata, uploadableMedia := uploadsrcdir.GetUploadableFiles(config)
// If there are no files to upload, do nothing
if len(uploadableMetadata) == 0 && len(uploadableMedia) == 0 {
fmt.Println("No uploadable files identified.")
return
}
// Ensure no files are too new too recent - if any are, say so and abort.
if uploadsrcdir.CheckAnyFileIsTooRecent(uploadableMedia) || uploadsrcdir.CheckAnyFileIsTooRecent(uploadableMetadata) {
fmt.Println("Some of the files have been updated too recently (1min). Blocking upload to prevent accidental uploads in the middle of an ongoing digitization effort.")
return
}
// Open WebDAV client
c := webdavupload.GetWebdavClient(config)
// Check that the remote is not currently occupied.
if webdavupload.CheckRemoteIsFree(c) == false {
fmt.Println("The remote is currently occupied (very recent files, or an import config is currently waiting to be processed, are present).")
return
}
// Upload the files
if len(uploadableMetadata) > 0 {
webdavupload.UploadMetadataFiles(c, os.Stdout, uploadableMetadata)
webdavupload.BatchUnlink(os.Stdout, uploadableMetadata)
}
if len(uploadableMedia) > 0 {
webdavupload.UploadMediaFiles(c, os.Stdout, uploadableMedia)
webdavupload.BatchUnlink(os.Stdout, uploadableMedia)
}
// Generate the import config.
fmt.Println("Generating and uploading import configuration")
webdavupload.SetImportConfigToTheRemote(c, config)
fmt.Println("DONE")
}

399
src/cli/cli.go Normal file
View File

@@ -0,0 +1,399 @@
package cli
import (
"bufio"
"fmt"
"os"
"strconv"
"strings"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/uploadsrcdir"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/webdavupload"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/meta"
)
// Interal wrapper around opening a new bufio.Reader and getting an input
// from it. As errors here likely really are something fundamentally wrong
// about the system, panic if there is one.
func getConsoleInput() string {
reader := bufio.NewReader(os.Stdin)
output, err := reader.ReadString('\n')
if err != nil {
fmt.Println("Failed to get console input!")
panic(err)
}
return strings.Trim(output, "\n\r")
}
// Prompts the user to enter a valid instance url until they do.
func requestInstanceLink() string {
fmt.Println("\nPlease enter URL of the instance of museum-digital you work in.")
fmt.Println("This should be a regional instance (e.g. hessen.museum-digital.de), not an aggregated one like nat.museum-digital.de.")
input := getConsoleInput()
output, validationErr := configloader.ValidateInstanceLink(input)
if validationErr != nil {
fmt.Println("This is not a valid instance of museum-digital / musdb. Please try again.")
fmt.Print("----\nThe specific error is:\n")
fmt.Println(validationErr)
fmt.Print("----\n\n")
return requestInstanceLink()
}
return output
}
// Prompts the user to enter a valid institution ID within the
// specified instance until they do.
func requestInstitutionId(instanceLink string) int {
fmt.Println("\nPlease enter ID of your institution.")
fmt.Println("You can find the ID of the institution ID in musdb. On the institution/museum editing page, you will find it as a large number below the orange bar \"museum\".")
input := getConsoleInput()
i, err := strconv.Atoi(input)
if err != nil {
fmt.Println("You did not enter a number. Please try again.")
return requestInstitutionId(instanceLink)
}
output, validationErr := configloader.ValidateInstitutionId(i, instanceLink)
if validationErr != nil {
fmt.Println("This is not a valid institution ID. Please try again.")
fmt.Print("----\nThe specific error is:\n")
fmt.Println(validationErr)
fmt.Print("----\n\n")
return requestInstitutionId(instanceLink)
}
return output
}
// Prompts the user to enter their username.
func requestUsername() string {
fmt.Println("\nPlease enter your user name in musdb (the name you use to log in with).")
input := getConsoleInput()
if input == "" {
fmt.Println("You may not enter an empty user name. Please try again.")
return requestUsername()
}
return input
}
// Prompts the user to enter a valid mail address until they do.
func requestMail() string {
fmt.Println("\nPlease enter the email you set for your musdb account.")
fmt.Println("Note that the spelling needs to be exactly the same as it is entered in musdb (e.g. in terms of capitalization).")
input := getConsoleInput()
output, validationErr := configloader.ValidateMail(input)
if validationErr != nil {
fmt.Println("This is not a valid mail address. Please try again.")
fmt.Print("----\nThe specific error is:\n")
fmt.Println(validationErr)
fmt.Print("----\n\n")
return requestMail()
}
return output
}
// Prompts the user to enter the webdav auth token.
func requestWebDavAuthToken() string {
fmt.Println("\nPlease enter your WebDAV authentication token.")
fmt.Println("You can generate a WebDAV authentication token on your account settings page in the tab \"WebDAV\".")
input := getConsoleInput()
if input == "" {
fmt.Println("You may not enter an empty WebDAV authentication token. Please try again.")
return requestWebDavAuthToken()
}
return input
}
// Prompts the user to enter a valid parser name until they do.
func requestParser() string {
fmt.Println("\nPlease enter the parser with which you want to run imports.")
fmt.Println("Parsers correspond with the input format. Available parsers are:")
fmt.Println("")
for _, parser := range(configloader.ListParsers()) {
fmt.Println("- " + parser.Title)
}
fmt.Println("")
input := getConsoleInput()
output, validationErr := configloader.ValidateParser(input)
if validationErr != nil {
fmt.Println("This is not a valid parser name. Please enter one from the list. Please try again.")
fmt.Print("----\nThe specific error is:\n")
fmt.Println(validationErr)
fmt.Print("----\n\n")
return requestParser()
}
return output
}
// Prompts the user to enter a valid folder for uploads.
func requestUploadFolder() string {
fmt.Println("\nPlease enter the full filepath of a directory in which the program should look for files to upload.")
fmt.Println("Note that the folder needs to exist.")
input := getConsoleInput()
output, validationErr := configloader.ValidateUploadDir(input)
if validationErr != nil {
fmt.Println("This is not a valid metadata folder. Please try again.")
fmt.Print("----\nThe specific error is:\n")
fmt.Println(validationErr)
fmt.Print("----\n\n")
return requestUploadFolder()
}
return output
}
// Prompts the user to say, whether they want to immediately publish the uploaded objects once imported.
func requestPublishOnImport() bool {
fmt.Println("\nDo you want to immediately publish your objects, once they have been imported? If so, enter 'y', else, enter 'n'")
input := getConsoleInput()
if input == "y" {
return true
} else if input == "n" {
return false
} else {
fmt.Println("You did not enter either y or n, please try again.")
return requestPublishOnImport()
}
}
// Wrapper around configloader.StoreConfigToFile that prints the error message
// if storing failed.
func storeConfigToFile(config configloader.MDWebDavUploaderConfig) {
err := configloader.StoreConfigToFile(config, "")
if err != nil {
fmt.Println("\nFailed to store config")
fmt.Println("Reason:")
fmt.Println(err)
panic("")
}
}
// Queries the user for each of the relevant values for setup.
func RunManualSetup() {
fmt.Println("Welcome to the setup of the museum-digital auto uploader")
fmt.Println("This is the fully manual version of the setup, hence you will be asked for each value")
config := configloader.MDWebDavUploaderConfig{}
config.InstanceLink = requestInstanceLink()
config.Username = requestUsername()
config.InstitutionId = requestInstitutionId(config.InstanceLink)
config.Mail = requestMail()
config.WebDavAuthToken = requestWebDavAuthToken()
config.Parser = requestParser()
config.UploadDir = requestUploadFolder()
config.PublishOnImport = requestPublishOnImport()
storeConfigToFile(config)
fmt.Println("Great! Your configuration has been stored. If you need to set further parser-specific settings, use --set-additional-setting. Else, use --upload.")
}
// Wrapper around configloader.LoadFromFile() that prints errors.
func loadConfigFromFile() configloader.MDWebDavUploaderConfig {
config, requiresSetup, err := configloader.LoadFromFile("")
// Re-run manual setup and then load anew.
if requiresSetup == true {
fmt.Println("Your config is incomplete. Please run the setup command before proceeding.")
RunManualSetup()
config, _, rerunerr := configloader.LoadFromFile("")
if rerunerr != nil {
fmt.Println("Another error occured in loading your configuration:")
fmt.Println(rerunerr)
panic("")
}
return config
} else if err != nil {
fmt.Println("Another error occured in loading your configuration:")
fmt.Println(err)
panic("")
}
return config
}
// Prints the current configuration.
func ShowCurrentConfig() {
config := loadConfigFromFile()
fmt.Println(config)
}
func SetAdditionalSetting() {
config := loadConfigFromFile()
// Welcome msg
fmt.Println("You are trying to set an additional setting. For this, you will need to enter 1) which setting you want to manipulate and 2) what you want to set it to.")
fmt.Println("-----\n")
// Print parser-specific settings
fmt.Println("Your currently configured parser is: " + config.Parser)
fmt.Println("You can learn more about the available settings for this parser in the parser documentation. Printing it now.")
for _, p := range(configloader.ListParsers()) {
if p.Title == config.Parser {
fmt.Println(p.Comment)
}
}
fmt.Println("\n-----\n")
// Get values
fmt.Println("Now, please enter which setting you would like to manipulate.")
settingKey := getConsoleInput()
fmt.Println("What would you like to set the setting to?")
settingValue := getConsoleInput()
if config.Settings == nil {
config.Settings = make(map[string]string)
}
config.Settings[settingKey] = settingValue
storeConfigToFile(config)
}
// Receives a list of files and an error. If the error exists, that is printed.
// Otherwise, the listed files of a directory are listed.
func printFolderContents(files []os.FileInfo, err error) {
if err != nil {
fmt.Println("Failed to list top level folder contents")
return
}
if len(files) == 0 {
fmt.Print("There are no files to print here.\n")
}
for _, f := range(files) {
fmt.Println(f.Name())
}
}
// Prints the top level folder contents of the WebDAV remote
func ListRemoteToplevel() {
config := loadConfigFromFile()
files, err := webdavupload.ListTopLevelContents(config)
printFolderContents(files, err)
}
// Prints the current contents of the remote metadata directory.
func ListRemoteMetadataDir() {
config := loadConfigFromFile()
files, err := webdavupload.ListMetadataDir(config)
printFolderContents(files, err)
}
// Prints the current contents of the remote media directory.
func ListRemoteMediaDir() {
config := loadConfigFromFile()
files, err := webdavupload.ListMediaDir(config)
printFolderContents(files, err)
}
// Prints the current contents of the local metadata directory.
func ListLocalMetadata() {
fmt.Println("Printing uploadable metadata files")
config := loadConfigFromFile()
files := uploadsrcdir.ListUploadableMetadata(config)
printFolderContents(files, nil)
}
// Prints the current contents of the local media directory.
func ListLocalMedia() {
fmt.Println("Printing uploadable media files")
config := loadConfigFromFile()
files := uploadsrcdir.ListUploadableMedia(config)
printFolderContents(files, nil)
}
// Prints current version.
func ShowVersion() {
fmt.Println(meta.GetVersion())
}
// Prints, if there is a current version.
func CheckVersionIsUpToDate() {
output, err := meta.CheckVersionIsUpToDate()
if err != nil {
fmt.Println("Failed to get RSS stream for most recent entry")
fmt.Print(err)
panic("Aborting")
}
// Empty output has no title
if output.Title == "" {
fmt.Println("Your version is up to date")
} else {
fmt.Println("There is a more recent version, version: " + output.Title)
fmt.Println("\nReleased: " + output.PubDate)
fmt.Println("\nGet it at " + output.Link)
}
}

View File

@@ -0,0 +1,28 @@
package configloader
import (
"net/http"
)
// Returns the HTTP status code for a url while sending requests using
// a user-agent specific to the app.
func HttpGetStatusCode(url string) (int, error) {
// create HTTP request
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return 0, err
}
// set User-Agent header
req.Header.Set("User-Agent", "museum-digital-uploader")
resp, err := http.DefaultClient.Do(req)
if err != nil {
return 0, err
}
defer resp.Body.Close()
return resp.StatusCode, nil
}

View File

@@ -0,0 +1,61 @@
package configloader
import _ "embed"
import (
"encoding/json"
"errors"
"slices"
)
type ParserListItem struct {
Title string `json:"title"`
Comment string `json:"comment"`
Authors []string `json:"authors"`
Links []string `json:"links"`
}
// Load parser list from JSON retrieved from musdb's API
// See https://<subdomain>.museum-digital.org/musdb/api/meta/list_import_parsers
// The embed module requires the variable to not be bound to a function.
//go:embed parser_list.json
var rawParserList []byte
// Returns a list of the available parsers.
func ListParsers() []ParserListItem {
parserListFromApi := []ParserListItem{}
// parserListFromApi := new(ApiParserListFormat)
json.Unmarshal(rawParserList, &parserListFromApi)
return parserListFromApi
}
// Checks if inputParser is in the list of available parsers.
// Returns the parser name or an error.
func ValidateParser(inputParser string) (string, error) {
parserList := ListParsers()
parserTitles := []string{}
for _, p := range(parserList) {
parserTitles = append(parserTitles, p.Title)
}
if slices.Contains(parserTitles, inputParser) {
return inputParser, nil
}
// Accepted invalid modes to clean up:
// ParserLido > Lido
// lido > Lido
cleaned := inputParser[6:]
if slices.Contains(parserTitles, cleaned) {
return cleaned, nil
}
return "", errors.New("Invalid parser selected")
}

View File

@@ -0,0 +1,55 @@
package configloader
import (
"testing"
)
// Tests that the list of parsers is not empty.
func TestListParserReturnsNonEmptyResult(t *testing.T) {
parserList := ListParsers()
if len(parserList) == 0 {
t.Fatalf("The list of returned parsers is empty")
}
}
// Ensures that validating a parser will not accept an invalid parser name.
func TestInvalidParserIsNotAccepted(t *testing.T) {
parser, err := ValidateParser("Liddasfasdo")
if parser != "" {
t.Fatalf("ValidateParser() should return an empty string + error")
}
if err == nil {
t.Fatalf("ValidateParser() did not return an error where it should")
}
}
// Ensures that a parser is valid at its expected name.
func TestValidParserIsAccepted(t *testing.T) {
parser, err := ValidateParser("Lido")
if parser != "Lido" {
t.Fatalf("Test expected validated parser for input 'Lido' to be 'Lido'")
}
if err != nil {
t.Fatalf("ValidateParser() returned an error where it should not")
}
}
// Prepending the prefix "Parser" to a parser name should work as well.
// In this case, the prefix should be removed by ValidateParser().
func TestValidParserIsAcceptedAndCleaned(t *testing.T) {
parser, err := ValidateParser("ParserLido")
if parser != "Lido" {
t.Fatalf("Test expected validated parser for input 'ParserLido' to be 'Lido'")
}
if err != nil {
t.Fatalf("ValidateParser() returned an error where it should not")
}
}

View File

@@ -0,0 +1,46 @@
package configloader
import (
"errors"
"net/url"
"strings"
)
// Validates a museum-digital instance URL to upload to.
// The link must be a valid URL. As it can be expected that people will
// enter broken values, the URL should be reduced to its hostname and
// rebuilt from there.
func ValidateInstanceLink(instanceUrl string) (string, error) {
parsed, err := url.Parse(instanceUrl)
if err != nil {
return "", err
}
// Check the hostname contains "museum-digital.de" or "museum-digital.org"
if !strings.HasSuffix(parsed.Host, "museum-digital.de") && !strings.HasSuffix(parsed.Host, "museum-digital.org") {
return "", errors.New("This tool only supports uploading to XYZ.museum-digital.de or XYZ.museum-digital.org")
}
// Check /musdb is available on the md instance
musdbUrl := url.URL{
Scheme: "https",
Host: parsed.Host,
Path: "/musdb",
}
statusCode, err := HttpGetStatusCode(musdbUrl.String())
if err != nil {
return "", err
}
if statusCode > 399 {
return "", errors.New("The museum-digital subdomain does not contain a musdb path")
}
// https should be assumed and enforced.
url := url.URL{
Scheme: "https",
Host: parsed.Host,
}
return url.String(), nil
}

View File

@@ -0,0 +1,69 @@
package configloader
import (
"testing"
)
// Test that ValidateInstanceLink() fails on non-URLs.
func TestValidateInstanceFailsOnNonUrl(t *testing.T) {
_, err := ValidateInstanceLink("abcmuseum-digital.org")
if err == nil {
t.Fatalf("ValidateInstanceLink() does not return an error on a non-URL")
}
}
// Test that ValidateInstanceLink() fails on links outside of md.
func TestValidateInstanceLinkFailsOutsideOfMd(t *testing.T) {
_, err := ValidateInstanceLink("test.museum-digital.de")
if err == nil {
t.Fatalf("ValidateInstanceLink() accepts URLs outside of the main museum-digital")
}
}
// Test that ValidateInstanceLink() fails on valid md aggregated instances
// (uploads can only happen to regional instances / those that have a dedicated
// instance of musdb).
func TestValidateInstanceLinkFailsWithAggregatedInstance(t *testing.T) {
_, err := ValidateInstanceLink("https://nat.museum-digital.de")
if err == nil {
t.Fatalf("ValidateInstanceLink() accepts URLs of aggregated instance")
}
}
// Test that ValidateInstanceLink() works with a valid instance of md.
func TestValidateInstanceWorks(t *testing.T) {
result, err := ValidateInstanceLink("https://hessen.museum-digital.de")
if err != nil {
t.Fatalf("ValidateInstanceLink() returns an error where it should work")
}
if result != "https://hessen.museum-digital.de" {
t.Fatalf("Output of ValidateInstanceLink() is not https://hessen.museum-digital.de where it should be")
}
}
// Test that ValidateInstanceLink() works with a valid instance of md and cleans paths.
func TestValidateInstanceDoesCleanPathFromUrl(t *testing.T) {
result, err := ValidateInstanceLink("https://hessen.museum-digital.de/home")
if err != nil {
t.Fatalf("ValidateInstanceLink() returns an error where it should work")
}
if result != "https://hessen.museum-digital.de" {
t.Fatalf("Output of ValidateInstanceLink() is not https://hessen.museum-digital.de where it should be")
}
}

View File

@@ -2,7 +2,6 @@ package configloader
import (
"errors"
"net/http"
"strconv"
)
@@ -16,17 +15,15 @@ func ValidateInstitutionId(institutionId int, instanceUrl string) (int, error) {
return 0, errors.New("Institution ID cannot be negative")
}
resp, err := http.Get(instanceUrl + "/institution/" + strconv.Itoa(institutionId))
statusCode, err := HttpGetStatusCode(instanceUrl + "/institution/" + strconv.Itoa(institutionId))
if err != nil {
return 0, err
}
if resp.StatusCode != 200 {
if statusCode != 200 {
return 0, errors.New("The institution page does not respond with HTTP 200, the institution does not seem to exist")
}
return institutionId, nil
}

View File

@@ -0,0 +1,15 @@
package configloader
import "net/mail"
// Validates an input mail address.
// Returns the valid mail address or an eQueries therror.
func ValidateMail(email string) (string, error) {
_, err := mail.ParseAddress(email)
if err != nil {
return "", err
}
return email, nil
}

View File

@@ -0,0 +1,27 @@
package configloader
import (
"testing"
)
// Test that ValidateMail() returns the mail address if it is valid.
func TestValidateMailAcceptsValidMailAddress(t *testing.T) {
_, err := ValidateMail("test@example.com")
if err != nil {
t.Fatalf("ValidateMail() returns an error on a valid mail address")
}
}
// Test that ValidateMail() filters out invalid mail addresses.
func TestValidateMailDoesNotAcceptInvalidMailAddress(t *testing.T) {
_, err := ValidateMail("test")
if err == nil {
t.Fatalf("ValidateMail() does not prevent entering an invalid mail address")
}
}

View File

@@ -1,67 +1,205 @@
package configloader
import (
"os"
"encoding/json"
"path/filepath"
"io/ioutil"
)
type MDWebDavUploaderConfig struct {
InstanceLink string `json:"instance"`
Mail string `json:"mail"`
WebDavAuthToken string `json:"token"`
InstitutionId int `json:"institution_id"`
Parser string `json:"parser"`
MetadataFolder string `json:"metadata_folder"`
MediaFolder string `json:"media_folder"`
PublishOnImport bool `json:"visible"`
InstanceLink string `json:"instance"`
Username string `json:"username"`
Mail string `json:"mail"`
WebDavAuthToken string `json:"token"`
InstitutionId int `json:"institution_id"`
Parser string `json:"parser"`
UploadDir string `json:"upload_directory"`
PublishOnImport bool `json:"visible"`
Settings map[string]string `json:"settings"`
}
type ParserListItem struct {
name string
// Returns the file path of the configuration file within the supplied
// directory. Moved to a dedicated function to provide a consistent
// filename.
func getConfigFileNameByDir(folder string) string {
return filepath.Join(folder, "config.json")
}
// Returns a uniform filepath for the configuration of this tool.
// Returns a uniform directory for the configuration of this tool.
// To be compatible across operating systems, this will be a JSON
// file in the same directory as the current programm.
func getConfigFilepath() string {
func getConfigFilepath() (string, error) {
// Get the OS-dependent configuration directory.
generalConfigDir, dirErr := os.UserConfigDir()
if dirErr != nil {
return "", dirErr
}
// Select a subdirectory of that directory to store application-specific
// settings in. Attempt to create it and return the config filepath as
// a file in that directory.
configDir := filepath.Join(generalConfigDir, "museum-digital-uploader")
generateConfigDirErr := os.Mkdir(configDir, 0700)
// The config directory could be created, return path of the configuration
// file in it.
if generateConfigDirErr == nil {
return getConfigFileNameByDir(configDir), nil // Use this to create files
}
// There has been an error creating the config directory.
// This may be either that the directory already exists, which is alright
// (and even expected on most runs). Or it might be, that the path is
// already occupied with a file. Or something else. In those cases,
// the error should be returned.
// If the path is already occupied, inspect it more closely.
if os.IsExist(generateConfigDirErr) {
info, err := os.Stat(configDir)
if err != nil {
return "", err
}
// The directory already exists and is a directory. This is fine,
// so we return the config filename in the existing config directory.
if info.IsDir() {
return getConfigFileNameByDir(configDir), nil // Use this to create files
}
}
return "", generateConfigDirErr
}
// Returns a uniform filepath for the cache for the list of
// available parsers.
func getParserCacheFilepath() string {
// Wrapper around getConfigFilepath() allowing for an override for testing.
func getConfigFilepathOrOverride(overridePath string) (string, error) {
// Override path is not validated, as it should only be used
// in test settings
if overridePath != "" {
return overridePath, nil
}
configFilePath, err := getConfigFilepath()
if err != nil {
return "", err
}
return configFilePath, nil
}
// Validates each of the values of MDWebDavUploaderConfig.
func ValidateConfig(conf MDWebDavUploaderConfig) (MDWebDavUploaderConfig, error) {
// Validate and clean instance link
instanceLink, instanceErr := ValidateInstanceLink(conf.InstanceLink)
if instanceErr != nil {
return conf, instanceErr
}
conf.InstanceLink = instanceLink
// Validate and clean mail
mailLink, mailErr := ValidateMail(conf.Mail)
if mailErr != nil {
return conf, mailErr
}
conf.Mail = mailLink
// Validate and clean institution ID
institutionIdLink, institutionIdErr := ValidateInstitutionId(conf.InstitutionId, conf.InstanceLink)
if institutionIdErr != nil {
return conf, institutionIdErr
}
conf.InstitutionId = institutionIdLink
// Validate and clean parser
parserLink, parserErr := ValidateParser(conf.Parser)
if parserErr != nil {
return conf, parserErr
}
conf.Parser = parserLink
// Validate and clean upload folder
uploadDir, mFolderErr := ValidateUploadDir(conf.UploadDir)
if mFolderErr != nil {
return conf, mFolderErr
}
conf.UploadDir = uploadDir
return conf, nil
}
// Loads configuration from the configuration file (located using
// getConfigFilepath()).
func LoadFromFile() MDWebDavUploaderConfig {
// The function parameter overridePath is only useful for test settings.
// Outside those, set an empty string to use the default config path.
// Output parameters:
// 1: Config
// 2: Requirement for a re-run of the setup
// 3: Error
func LoadFromFile(overridePath string) (MDWebDavUploaderConfig, bool, error) {
// Get config file path
configFile, configFileErr := getConfigFilepathOrOverride(overridePath)
if configFileErr != nil {
return MDWebDavUploaderConfig{}, false, configFileErr
}
// Read the file
configFileBytes, ioErr := ioutil.ReadFile(configFile)
if ioErr != nil {
return MDWebDavUploaderConfig{}, true, ioErr
}
// Parse the file into a configuration struct
var data MDWebDavUploaderConfig
unmarshalErr := json.Unmarshal(configFileBytes, &data)
if unmarshalErr != nil {
return MDWebDavUploaderConfig{}, true, ioErr
}
// Validate data - the configuration file may have been altered externally
config, validationErr := ValidateConfig(data)
if validationErr != nil {
return MDWebDavUploaderConfig{}, true, validationErr
}
// Config could be parsed
return config, false, nil
}
// Stores a parser list to a cache file (located via
// getParserCacheFilepath()).
func StoreParserList(parserList []ParserListItem) {
}
// Returns a list of the available parsers.
func ListParsers() []ParserListItem {
}
// Checks if inputParser is in the list of available parsers.
// Returns the parser name or an error.
func ValidateParser(inputParser string) (string, error) {
}
// Validates an input mail address.
// Returns the valid mail address or an eQueries therror.
func ValidateMail(mail string) (string, error) {
}
// Validates an entered instance URL.
// Returns the entered URL in a unified form (minus optional
// trailing slashes and the /musdb suffix) or an error.
func ValidateInstanceLink(instanceUrl string) (string, error) {
// Stores a config from MDWebDavUploaderConfig to the config file.
// As with LoadFromFile(), it is possible to provide an overridePath for testing.
func StoreConfigToFile(conf MDWebDavUploaderConfig, overridePath string) error {
config, validationErr := ValidateConfig(conf)
if validationErr != nil {
return validationErr
}
configFilePath, configFileErr := getConfigFilepathOrOverride(overridePath)
if configFileErr != nil {
return configFileErr
}
configJson, encodeErr := json.Marshal(config)
if encodeErr != nil {
return encodeErr
}
writeErr := ioutil.WriteFile(configFilePath, configJson, 0644)
if writeErr != nil {
return writeErr
}
return nil
}

View File

@@ -0,0 +1,173 @@
package configloader
import (
"testing"
"os"
"path/filepath"
)
// Returns a generally valid config. Single values can then be
// replaced with the actually tested contents.
func getTestConfig() MDWebDavUploaderConfig {
input := MDWebDavUploaderConfig{}
input.InstanceLink = "https://hessen.museum-digital.de/home"
input.Mail = "test@example.com"
input.InstitutionId = 1;
input.Parser = "Lido"
tmpDir := os.TempDir()
testDir := filepath.Join(tmpDir, "/existing-dir-for-import")
mkdirErr := os.MkdirAll(testDir, os.ModePerm)
if mkdirErr != nil {
panic("Test failure: Failed to create test dir")
}
input.UploadDir = testDir
return input
}
// Test that ValidateConfig() fails on non-URLs.
func TestValidateUploaderConfigFailsOnInvalidInstanceUrl(t *testing.T) {
input := getTestConfig()
input.InstanceLink = "abcmuseum-digital.org"
_, err := ValidateConfig(input)
if err == nil {
t.Fatalf("ValidateConfig() does not return an error on a non-URL (via ValidateInstanceLink)")
}
}
// Test that ValidateConfig cleans an instance URL as per ValidateInstanceLink().
func TestValidateUploaderConfigCleansInstanceViaValidateInstanceLink(t *testing.T) {
input := getTestConfig()
input.InstanceLink = "https://hessen.museum-digital.de/home"
returnVal, _ := ValidateConfig(input)
if returnVal.InstanceLink != "https://hessen.museum-digital.de" {
t.Fatalf("Failed to clean up input URL")
}
}
// Test that ValidateConfig() fails on invalid mail.
func TestValidateUploaderConfigFailsOnInvalidMail(t *testing.T) {
input := getTestConfig()
input.Mail = "test"
_, err := ValidateConfig(input)
if err == nil {
t.Fatalf("ValidateConfig() does not return an error on a mail address")
}
}
// Test that ValidateConfig() accepts a valid mail address.
func TestValidateUploaderConfigAcceptsValidMail(t *testing.T) {
input := getTestConfig()
input.Mail = "test@example.com"
returnVal, _ := ValidateConfig(input)
if returnVal.Mail != "test@example.com" {
t.Fatalf("Failed to accept valid mail")
}
}
// Test that ValidateConfig() fails on negative / invalid IDs.
func TestValidateUploaderConfigFailsOnInvalidInstitutionId(t *testing.T) {
input := getTestConfig()
input.InstitutionId = -1
_, err := ValidateConfig(input)
if err == nil {
t.Fatalf("ValidateConfig() does not return an error on an invalid institution ID")
}
}
// Test that ValidateConfig() accepts valid institution IDs.
func TestValidateUploaderConfigAcceptsValidInstitutionId(t *testing.T) {
input := getTestConfig()
input.InstitutionId = 1
returnVal, _ := ValidateConfig(input)
if returnVal.InstitutionId != 1 {
t.Fatalf("Failed to accept valid institution ID")
}
}
// Test that ValidateConfig() fails on negative / invalid IDs.
func TestValidateUploaderConfigFailsOnInvalidParser(t *testing.T) {
input := getTestConfig()
input.Parser = "nonexistentparser"
_, err := ValidateConfig(input)
if err == nil {
t.Fatalf("ValidateConfig() does not return an error on an invalid parser")
}
}
// Test that ValidateConfig() accepts valid ParserDs.
func TestValidateUploaderConfigAcceptsValidParser(t *testing.T) {
input := getTestConfig()
input.Parser = "ParserLido"
returnVal, _ := ValidateConfig(input)
if returnVal.Parser != "Lido" {
t.Fatalf("Failed to accept and clean valid parser")
}
}
// Test that ValidateConfig() fails on non-existent folder.
func TestValidateUploaderConfigFailsOnInvalidUploadDir(t *testing.T) {
input := getTestConfig()
input.UploadDir = "nonexistentfolder"
_, err := ValidateConfig(input)
if err == nil {
t.Fatalf("ValidateConfig() does not return an error on an invalid upload folder")
}
}
// Test that saving and reading config works.
func TestWritingAndReadingConfigWorks(t *testing.T) {
input := getTestConfig()
input, _ = ValidateConfig(input)
writeErr := StoreConfigToFile(input, input.UploadDir + "/config.json")
if writeErr != nil {
t.Log("Error:")
t.Log(writeErr)
t.Fatalf("Failed to write config to override path")
}
loadedFromFile, setupRequired, err := LoadFromFile(input.UploadDir + "/config.json")
if setupRequired != false {
t.Fatalf("Expected no setup to be required, but return value indicated thus")
}
if err != nil {
t.Fatalf("Returned an error on trying to load config file")
}
// Golang can't compare structs with slices or maps
if input.InstanceLink != loadedFromFile.InstanceLink || input.Mail != loadedFromFile.Mail || input.WebDavAuthToken != loadedFromFile.WebDavAuthToken || input.InstitutionId != loadedFromFile.InstitutionId || input.Parser != loadedFromFile.Parser || input.UploadDir != loadedFromFile.UploadDir || input.PublishOnImport != loadedFromFile.PublishOnImport {
t.Log("Input")
t.Log(input)
t.Log("Loaded output")
t.Log(loadedFromFile)
t.Fatalf("Failed to write and then load the same config")
}
}

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,20 @@
package importconfiggen
import (
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
)
func GenerateImportConfig(config configloader.MDWebDavUploaderConfig) string {
output := "# Generated by user: " + config.Username
output += "\nmail: " + config.Mail + "\nparser: " + config.Parser
if config.Settings != nil {
for settingKey, settingValue := range(config.Settings) {
output += "\nsetting: " + settingKey + ": " + settingValue
}
}
return output
}

View File

@@ -0,0 +1,47 @@
package meta
import (
"net/http"
"encoding/xml"
)
// Very reduced item, as only title, link and pubDate are useful for a version check.
type Item struct {
Title string `xml:"title"`
Link string `xml:"link"`
PubDate string `xml:"pubDate"`
}
type Channel struct {
Items []Item `xml:"item"`
}
type Rss struct {
Channel Channel `xml:"channel"`
}
// Checks if the current version is up to date or not. If there is a newer / different
// version.
func CheckVersionIsUpToDate() (Item, error) {
resp, err := http.Get("https://gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/releases.rss")
if err != nil {
return Item{}, err
}
defer resp.Body.Close()
rss := Rss{}
decoder := xml.NewDecoder(resp.Body)
decodeErr := decoder.Decode(&rss)
if decodeErr != nil {
return Item{}, err
}
newestItem := rss.Channel.Items[0];
if (newestItem.Title == GetVersion() || newestItem.Title == "v" + GetVersion()) {
return Item{}, nil
}
return newestItem, nil
}

6
src/meta/meta.go Normal file
View File

@@ -0,0 +1,6 @@
package meta
// Returns current version number.
func GetVersion() string {
return "0.1.1"
}

View File

@@ -0,0 +1,130 @@
package uploadsrcdir
import (
"path/filepath"
"fmt"
"io/fs"
"os"
"runtime"
"slices"
"strings"
"sync"
"time"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
)
var MetadataFileExts = []string{".xml", ".csv", ".json"}
var MediaFileExts = []string{".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tif", ".tiff", ".pdf", ".mp3", ".mp4"}
// Lists all files in a folder that are of any of the expected file extensions.
func getFilesInDir(startPath string, fileExts []string) []os.FileInfo {
allFiles := []os.FileInfo{}
filepath.WalkDir(startPath, func(path string, entry fs.DirEntry, err error) error {
if err != nil {
return err
}
if slices.Contains(fileExts, filepath.Ext(entry.Name())) || slices.Contains(fileExts, strings.ToLower(filepath.Ext(entry.Name()))) {
// add entry and it's size to dirMap
file, err := entry.Info()
if err != nil {
fmt.Println("Error retrieving file info for:", err)
} else {
allFiles = append(allFiles, file)
}
}
return nil
})
return allFiles
}
// Lists all files to upload in metadata directory
func ListUploadableMetadata(config configloader.MDWebDavUploaderConfig) []os.FileInfo {
return getFilesInDir(config.UploadDir, MetadataFileExts);
}
// Lists all files to upload in metadata directory
func ListUploadableMedia(config configloader.MDWebDavUploaderConfig) []os.FileInfo {
return getFilesInDir(config.UploadDir, MediaFileExts);
}
// Returns lists of uploadable folder contents grouped into a) metadata files and b) media files.
func GetUploadableFiles(config configloader.MDWebDavUploaderConfig) ([]string, []string) {
metadataFiles := []string{}
mediaFiles := []string{}
filepath.WalkDir(config.UploadDir, func(path string, entry fs.DirEntry, err error) error {
if err != nil {
return err
}
if slices.Contains(MetadataFileExts, filepath.Ext(entry.Name())) || slices.Contains(MetadataFileExts, strings.ToLower(filepath.Ext(entry.Name()))) {
metadataFiles = append(metadataFiles, path)
}
if slices.Contains(MediaFileExts, filepath.Ext(entry.Name())) || slices.Contains(MediaFileExts, strings.ToLower(filepath.Ext(entry.Name()))) {
mediaFiles = append(mediaFiles, path)
}
return nil
})
return metadataFiles, mediaFiles
}
// Checks if a list of files have been modified in the last minute.
// If so, returns true.
func CheckAnyFileIsTooRecent(files []string) bool {
now := time.Now()
maxConcTasks := min(10, runtime.NumCPU())
// Set a semaphore to restrict the number of concurrent upload tasks.
semaphore := make(chan struct{}, maxConcTasks)
wg := &sync.WaitGroup{}
output := false
for _, f := range(files) {
semaphore <- struct{}{} // acquire
wg.Add(1)
go func() {
defer wg.Done()
file, fStatErr := os.Stat(f)
if fStatErr != nil {
panic("Failed to read file: " + f)
}
if diff := now.Sub(file.ModTime()); diff < 1 * time.Minute {
print("File " + file.Name() + " has been changed within the last minute\n")
output = true
}
<-semaphore // release
}()
}
wg.Wait()
return output
}

View File

@@ -0,0 +1,115 @@
package webdavupload
import (
"os"
"sync"
"time"
"github.com/studio-b12/gowebdav"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
)
// Takes a path on the webdav remote and reads folder contents from it.
func listWebDavFolderContents(config configloader.MDWebDavUploaderConfig, path string) ([]os.FileInfo, error) {
c := gowebdav.NewClient(getWebDavHost(config), config.Username, config.WebDavAuthToken)
c.Connect()
files, err := c.ReadDir(path)
return files, err
}
// Returns a list of the files on the top level of the remote.
func ListTopLevelContents(config configloader.MDWebDavUploaderConfig) ([]os.FileInfo, error) {
return listWebDavFolderContents(config, ".")
}
// Returns a list of the files in the current metadata dir of the remote.
func ListMetadataDir(config configloader.MDWebDavUploaderConfig) ([]os.FileInfo, error) {
return listWebDavFolderContents(config, "./IMPORT_XML")
}
// Returns a list of the files in the current media dir of the remote.
func ListMediaDir(config configloader.MDWebDavUploaderConfig) ([]os.FileInfo, error) {
return listWebDavFolderContents(config, "./IMPORT_IMG")
}
func checkImportConfigExists(c *gowebdav.Client) bool {
_, err := c.Stat("./import_config.txt")
if gowebdav.IsErrNotFound(err) == true {
return false
}
return true
}
// Checks if a remote directory is free for updating:
// This means it can be read, and it does not contain files more recent than
// 10 minutes.
func checkRemoteDirIsFree(c *gowebdav.Client, path string) bool {
now := time.Now()
// Check metadata dir
files, err := c.ReadDir(path)
if err != nil {
print("Failed to load remote contents")
return false
}
for _, f := range(files) {
if diff := now.Sub(f.ModTime()); diff < 10 * time.Minute {
print("File " + f.Name() + " has been recently renamed")
return false
}
}
return true
}
// Checks if either the metadata or the image folder contains files that have
// been changed in the last 10 minutes.
// In this case, it can be assumed that there is a concurrent upload taking place.
func checkRemoteImportFilesAreTooRecent(c *gowebdav.Client) bool {
var wg sync.WaitGroup
var metadataDirFree bool
var mediaDirFree bool
wg.Add(2)
go func() {
defer wg.Done()
metadataDirFree = checkRemoteDirIsFree(c, "./IMPORT_XML")
}()
go func() {
defer wg.Done()
mediaDirFree = checkRemoteDirIsFree(c, "./IMPORT_IMG")
}()
wg.Wait()
if metadataDirFree == true && mediaDirFree == true {
return false
}
return true
}
// Checks the remote for the existence of an import_config.txt.
func CheckRemoteIsFree(c *gowebdav.Client) bool {
if checkImportConfigExists(c) == true {
return false
}
// Check if the remote media and metadata directories currently
// contain very recent files indicating a concurrent upload.
if checkRemoteImportFilesAreTooRecent(c) == true {
return false
}
return true
}

View File

@@ -0,0 +1,35 @@
package webdavupload
import (
"net/url"
"strconv"
"github.com/studio-b12/gowebdav"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
)
// Generates the path for a webdav endpoint for the configured instance and institution.
func getWebDavHost(config configloader.MDWebDavUploaderConfig) string {
parsed, err := url.Parse(config.InstanceLink)
if err != nil {
panic("Your settings must be broken. Failed to load instance URL.")
}
output := url.URL{
Scheme: "https",
Host: parsed.Host,
Path: "/musdb/webdav/" + strconv.Itoa(config.InstitutionId),
}
return output.String()
}
// Returns a WebDAV client for the current config.
func GetWebdavClient(config configloader.MDWebDavUploaderConfig) *gowebdav.Client {
c := gowebdav.NewClient(getWebDavHost(config), config.Username, config.WebDavAuthToken)
c.Connect()
return c
}

132
src/webdavupload/Upload.go Normal file
View File

@@ -0,0 +1,132 @@
package webdavupload
import (
"fmt"
"path/filepath"
"os"
"io"
"net/http"
"runtime"
"sync"
"sync/atomic"
"github.com/studio-b12/gowebdav"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/importconfiggen"
)
// Writes an import config to the remote
func SetImportConfigToTheRemote(c *gowebdav.Client, config configloader.MDWebDavUploaderConfig) error {
importConf := importconfiggen.GenerateImportConfig(config)
return c.Write("import_config.txt", []byte(importConf), 0660)
}
// Uploads a list of files to the target folder.
func uploadFiles(c *gowebdav.Client, w io.Writer, files []string, remoteTarget string, outputContext string) {
// Check if the io.Writer is a http writer
_, wImplementsHttpFlusher := interface{}(w).(http.Flusher)
total := len(files)
var counter atomic.Uint64
// Determine the number of upload tasks to be processed concurrently.
// 10 will be a hard maximum to not spam the server.
maxConcTasks := min(10, runtime.NumCPU())
// Set a semaphore to restrict the number of concurrent upload tasks.
semaphore := make(chan struct{}, maxConcTasks)
wg := &sync.WaitGroup{}
fmt.Fprintf(w, "Will upload %v files. Processing %v tasks at a time.\n", total, maxConcTasks)
if wImplementsHttpFlusher == true {
w.(http.Flusher).Flush()
}
for _, f := range(files) {
semaphore <- struct{}{} // acquire
wg.Add(1)
go func() {
defer wg.Done()
basename := filepath.Base(f)
file, fOpenErr := os.Open(f)
if fOpenErr != nil {
panic("Failed to read file: " + f)
}
defer file.Close()
c.WriteStream("./" + remoteTarget + "/" + basename, file, 0644)
counter.Add(1)
fmt.Fprintf(w, "Uploading %d of %d - File: %s (%s)\n", counter, total, basename, outputContext)
if wImplementsHttpFlusher == true {
w.(http.Flusher).Flush()
}
<-semaphore // release
}()
}
wg.Wait()
fmt.Println("Done")
}
// Uploads the selected metadata files.
func UploadMetadataFiles(c *gowebdav.Client, w io.Writer, files []string) {
uploadFiles(c, w, files, "IMPORT_XML", "Uploading metadata files")
}
// Uploads the selected media files.
func UploadMediaFiles(c *gowebdav.Client, w io.Writer, files []string) {
uploadFiles(c, w, files, "IMPORT_IMG", "Uploading media files")
}
// Removes a list of files.
func BatchUnlink(w io.Writer, files []string) {
_, wImplementsHttpFlusher := interface{}(w).(http.Flusher)
maxConcTasks := min(10, runtime.NumCPU())
// Set a semaphore to restrict the number of concurrent upload tasks.
semaphore := make(chan struct{}, maxConcTasks)
wg := &sync.WaitGroup{}
for _, f := range(files) {
semaphore <- struct{}{} // acquire
wg.Add(1)
go func() {
defer wg.Done()
err := os.Remove(f)
if err != nil {
panic("Failed to delete file " + f)
}
fmt.Fprintf(w, "Delete file %s\n", f)
<-semaphore // release
}()
}
wg.Wait()
if wImplementsHttpFlusher == true {
w.(http.Flusher).Flush()
}
}

View File

@@ -0,0 +1,25 @@
<!DOCTYPE html>
<html>
<head>
<title>museum-digital uploader</title>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8"/>
<link rel="shortcut icon" sizes="16x16 32x32" href="/logo.svg" />
<link rel="shortcut icon" sizes="64x64" href="/logo.svg" />
<link rel="apple-touch-icon" sizes="256x256" href="/logo.svg" />
<meta name="theme-color" content="#FFF" />
<link rel="stylesheet" type="text/css" href="/md-uploader.css" />
</head>
<body>
<header>
<img src="/logo.svg">
<h1>museum-digital:uploader</h1>
<span id="actions"></span>
</header>
<div id="wrap">
<script type='text/javascript' src='/md-uploader.js' async></script>
</body>
</html>

View File

@@ -0,0 +1,423 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
width="1020"
height="1020"
id="svg2985"
version="1.1"
inkscape:version="1.2 (dc2aedaf03, 2022-05-15)"
sodipodi:docname="logo-md-code-black.svg"
inkscape:export-filename="mdlogo-code-512px.png"
inkscape:export-xdpi="48.189999"
inkscape:export-ydpi="48.189999"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/">
<title
id="title42">General logo for museum-digital (black background)</title>
<defs
id="defs2987" />
<sodipodi:namedview
id="base"
pagecolor="#ffffff"
bordercolor="#ffffff"
borderopacity="0.0"
inkscape:pageopacity="0.0"
inkscape:pageshadow="0"
inkscape:zoom="0.98994949"
inkscape:cx="375.27167"
inkscape:cy="658.11439"
inkscape:document-units="px"
inkscape:current-layer="layer1"
showgrid="false"
inkscape:window-width="3840"
inkscape:window-height="2076"
inkscape:window-x="0"
inkscape:window-y="-33"
inkscape:window-maximized="1"
fit-margin-top="5"
fit-margin-left="5"
fit-margin-right="5"
fit-margin-bottom="5"
showguides="false"
inkscape:showpageshadow="2"
inkscape:pagecheckerboard="0"
inkscape:deskcolor="#d1d1d1" />
<metadata
id="metadata2990">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<cc:license
rdf:resource="http://creativecommons.org/licenses/by/4.0/" />
<dc:title>General logo for museum-digital (black background)</dc:title>
<dc:date>2019</dc:date>
<dc:creator>
<cc:Agent>
<dc:title>museum-digital</dc:title>
</cc:Agent>
</dc:creator>
</cc:Work>
<cc:License
rdf:about="http://creativecommons.org/licenses/by/4.0/">
<cc:permits
rdf:resource="http://creativecommons.org/ns#Reproduction" />
<cc:permits
rdf:resource="http://creativecommons.org/ns#Distribution" />
<cc:requires
rdf:resource="http://creativecommons.org/ns#Notice" />
<cc:requires
rdf:resource="http://creativecommons.org/ns#Attribution" />
<cc:permits
rdf:resource="http://creativecommons.org/ns#DerivativeWorks" />
</cc:License>
</rdf:RDF>
</metadata>
<g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(8,-40.362183)">
<rect
style="fill:#000000;fill-opacity:1;stroke:#000000;stroke-width:8.82220364;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect845"
width="1011.1779"
height="1011.1778"
x="-3.5888982"
y="44.773335"
ry="0" />
<flowRoot
xml:space="preserve"
id="flowRoot3763"
style="font-style:normal;font-weight:normal;line-height:0.01%;font-family:sans-serif;text-align:end;letter-spacing:0px;word-spacing:0px;text-anchor:end;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-opacity:1"
transform="matrix(1.0379746,0,0,1.0379746,-169.56958,-12.773255)"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997"><flowRegion
id="flowRegion3765"
style="font-family:sans-serif;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-opacity:1"><rect
id="rect3767"
width="1008.1323"
height="1000.051"
x="-6.0609155"
y="50.290859"
style="font-family:sans-serif;text-align:end;text-anchor:end;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-opacity:1" /></flowRegion><flowPara
id="flowPara3769"
style="font-size:40px;line-height:1.25;font-family:sans-serif;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-opacity:1"> </flowPara></flowRoot>
<text
xml:space="preserve"
style="font-style:normal;font-variant:normal;font-stretch:normal;font-size:12.4557px;line-height:0%;font-family:'Bookman Old Style';-inkscape-font-specification:'Bookman Old Style Semi-Light';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:1.03797;stroke-opacity:1"
x="348.29947"
y="659.89093"
id="text3771"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833"
width="79.41317"
height="79.41317"
x="98.490639"
y="742.45844"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-3"
width="79.413177"
height="79.41317"
x="98.490639"
y="626.96552"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6"
width="79.41317"
height="79.41317"
x="98.490639"
y="511.47241"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:none;fill-opacity:1;stroke:#ffffff;stroke-width:10.21611977;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-3-7"
width="82.178253"
height="47.530357"
x="97.108063"
y="429.24481"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-5"
width="79.413162"
height="79.41317"
x="271.73004"
y="742.45844"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-3-3"
width="79.41317"
height="79.41317"
x="271.73004"
y="626.96552"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5"
width="79.413162"
height="79.41317"
x="271.73004"
y="511.47241"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-5-6"
width="79.413162"
height="79.41317"
x="444.96948"
y="742.45844"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-3-3-2"
width="79.41317"
height="79.41317"
x="444.96948"
y="626.96545"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9"
width="79.413162"
height="79.41317"
x="444.96948"
y="511.47235"
rx="0"
ry="39.706581"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:10.21612072;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-1"
width="47.530357"
height="82.178246"
x="201.05171"
y="510.08987"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:10.21612072;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-1-2"
width="47.530357"
height="82.178246"
x="374.29114"
y="510.08987"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7"
width="79.413162"
height="79.41317"
x="595.11035"
y="742.45844"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-0"
width="79.413162"
height="79.41317"
x="710.60327"
y="742.45844"
rx="0"
ry="0"
inkscape:transform-center-x="-9.9165682"
inkscape:transform-center-y="47.832846"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-9"
width="79.413162"
height="79.41317"
x="826.09625"
y="742.45844"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-3"
width="79.413162"
height="79.41317"
x="595.11035"
y="626.96552"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-3-6"
width="79.413162"
height="79.41317"
x="595.11035"
y="511.47241"
rx="0"
ry="39.706581"
inkscape:transform-center-x="-29.166407"
inkscape:transform-center-y="139.41528"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-3-6-0"
width="79.413162"
height="79.41317"
x="710.60327"
y="511.47241"
rx="0"
ry="0"
inkscape:transform-center-x="-29.166375"
inkscape:transform-center-y="139.41528"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:none;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-6"
width="79.413162"
height="79.41317"
x="826.09625"
y="626.96552"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-3-2"
width="79.413162"
height="79.41317"
x="826.09625"
y="511.47241"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-3-6-6"
width="79.413162"
height="79.41317"
x="826.09625"
y="395.97949"
rx="0"
ry="0"
inkscape:transform-center-x="-29.166404"
inkscape:transform-center-y="139.4153"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:12.98119068;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-3-6-6-1"
width="79.413162"
height="79.41317"
x="826.09625"
y="280.48651"
rx="0"
ry="0"
inkscape:transform-center-x="-29.166404"
inkscape:transform-center-y="139.41528"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:9.61340618;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-3"
width="41.781116"
height="82.780952"
x="443.28558"
y="509.78845"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:9.61340523;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-75"
width="82.780945"
height="41.78112"
x="443.28558"
y="550.78833"
rx="0"
ry="0"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:9.61340523;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-3-6-3"
width="82.780945"
height="41.78112"
x="593.42645"
y="550.78839"
rx="0"
ry="0"
inkscape:transform-center-x="-30.4033"
inkscape:transform-center-y="73.349637"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
<rect
style="fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:9.61340523;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect833-6-5-9-7-3-6-5"
width="41.781116"
height="82.780952"
x="634.42627"
y="509.78851"
rx="0"
ry="0"
inkscape:transform-center-x="-15.345135"
inkscape:transform-center-y="145.32765"
inkscape:export-xdpi="96.379997"
inkscape:export-ydpi="96.379997" />
</g>
</svg>

After

Width:  |  Height:  |  Size: 15 KiB

View File

@@ -0,0 +1,183 @@
:root {
--color-bg: #FFF;
--color-bg2: #F5F5F5;
--color-fg: #000;
--color-fg2: #212121;
--color-bg-dark: #212121;
--color-fg-medium: #484848;
--color-borders: #D6D6D6;
--color-accent: #546E7A;
--color-accent-hover: #455A64;
--color-accent-focus: #78909C;
--def-padding-main: 1em;
--def-border-radius: 6px;
}
@media (prefers-color-scheme: dark) {
:root {
--color-bg: #000;
--color-bg2: #2685AF;
--color-fg: #FFF;
--color-fg2: #D6D6D6;
--color-bg-dark: #212121;
--color-fg-medium: #BDBDBD;
--color-borders: #424242;
--color-accent: #546E7A;
--color-accent-hover: #90A4AE;
--color-accent-focus: #78909C;
}
}
html {
background:
linear-gradient(135deg,#0000 18.75%,var(--color-bg2) 0 31.25%,#0000 0),
repeating-linear-gradient(45deg,var(--color-bg2) -6.25% 6.25%,var(--color-bg) 0 18.75%);
background-size: 64px 64px;
color: var(--color-fg);
padding: 0;
font-family: Sans;
}
body { margin: 0; padding: 0; }
header { display: grid; grid-template-columns: auto auto 100fr auto;
align-items: center;
background: var(--color-bg); font-size: 1.5em; padding: .4em 2rem;
box-shadow: 0 4px 2px -2px var(--color-borders); }
header * { margin: 0; padding: 0; vertical-align: middle; }
header img { height: 1em; }
header #actions { grid-column: 4; font-size: .85rem; font-weight: bold; }
header #actions > * { padding: .2em .5em;
border-radius: .2em; color: var(--color-fg2); transition: .2s; }
header #actions > :hover { background: var(--color-accent-hover); color: var(--color-bg); }
h1 { display: inline-block; margin-left: 1em; font-size: .8em; }
#wrap { display: grid;
grid-template-columns: 450px 1fr; grid-template-rows: auto auto;
margin: 1em 2rem; grid-gap: 2em; }
#wrap > * { vertical-align: top; }
main,
aside,
.sec-aside { background: var(--color-bg); border-radius: 1px solid var(--color-borders); box-shadow: 0 0 6px var(--color-borders); border-radius: 2px; }
aside,
.sec-aside { grid-column: 1; }
main { grid-column: 2; grid-row: 1 / span 2; }
h2 { margin: 0; padding: .8rem; font-size: 1.4em;
border-bottom: .15em solid var(--color-borders); }
h3 { margin: 0; padding: .8rem; border-bottom: .15em solid var(--color-borders); }
input { background: var(--color-bg); color: var(--color-fg); }
aside .label-input { display: grid;
grid-template-columns: 150px auto 2em; border-bottom: 1px solid var(--color-borders); }
aside .label-input:hover { border-color: var(--color-accent); }
aside .label-input > * { display: inline-block;
font-size: .9em;
padding: .5rem .8rem; border: 0; outline: 0; }
aside label { font-weight: bold; color: var(--color-fg-medium); }
aside .label-input button { background: none; transition: background .2s; }
aside .label-input button:hover { background: var(--color-accent); }
.icon-help:before { content: " ? "; font-weight: bold; font-size: .8em;
text-align: center; color: var(--color-fg2); transition: background .2s; }
.icon-help { transition: background .2s; }
.icon-help:hover { background: var(--color-accent); }
select { width: 100%; background: var(--color-bg); color: var(--color-fg); }
.updateB { display: block; width: calc(100% - 1em); background: var(--color-bg);
margin: .5em; padding: .5em;
text-align: center;
border: 1px solid var(--color-bg); border-radius: 2px; font-weight: bold;
color: var(--color-accent); transition: background .2s, color .2s;}
.updateB:hover { background: var(--color-accent-hover); color: var(--color-bg); }
.upload-file-list > span { display: block;
padding: .3em .8rem;
font-family: Mono, Courier; font-size: .9em;
border-bottom: 1px solid var(--color-borders); transition: border .2s, background .2s; }
.upload-file-list > span:hover { background: var(--color-bg2); border-color: var(--color-accent-hover); }
.upload-file-list:empty { display: block; position: relative; height: 5em; }
.upload-file-list:empty:before { content: " - - - - - ";
position: absolute; left: 50%; top: 50%; transform: translate(-50%, -50%);
font-weight: bold; font-size: 3em; color: var(--color-borders); }
main { position: relative; }
main h3 { position: sticky; top: 0; background: var(--color-bg); }
.transfer-overlay { display: block;
position: fixed; left: 5em; bottom: 0; width: calc(100% - 10em);
background: var(--color-bg);
border: 2px solid var(--color-accent-hover); border-bottom-width: 0;
box-shadow: 0 0 6px var(--color-borders);
z-index: 10; animation: slide-up .4s ease-out; }
.transfer-overlay h3 { border-bottom: none; }
.transfer-overlay-hl { display: grid; grid-template-columns: auto 1fr auto;
padding-right: .8em;
align-items: center; border-bottom: .15em solid var(--color-borders); }
.transfer-overlay-hl .overlay-close { display: inline-block;
color: var(--color-fg2);
padding: .5em; border-radius: 2px;
grid-column: 3;
transition: .2s; }
.transfer-overlay-hl .overlay-close:hover { background: var(--color-accent-hover); color: var(--color-bg); }
.transfer-overlay-hl .overlay-close:before { display: inline-block; content: " x ";}
.transfer-msgs { font-size: .9em; }
.transfer-msgs > * { display: block; border-bottom: 1px solid var(--color-borders); transition: .2s; }
.transfer-msgs > :hover { border-bottom-color: 1px solid var(--color-accent-hover); }
@keyframes slide-up {
from { transform: translateY(100%); }
to { transform: initial; }
}
iframe { display: block; width: 100%; border: none; height: 60vh; }
.parser-desc { display: block; padding: 0 .8rem;
font-size: .85em; white-space: pre-wrap; }
#wrap.setup-required { grid-template-columns: 1fr 450px 1fr; justify-content: center; }
.setup-required aside { grid-column: 2; width: 450px; margin: auto; }
.setup-required main { display: none; }
/***
* Slider
* Thanks: mrhyddenn
* The following CSS slider is licensed under the MIT license, 2025 mrhyddenn.
* https://uiverse.io/mrhyddenn/old-fish-66
* */
/* The switch - the box around the slider */
.switch { font-size: 17px; position: relative; display: inline-block; width: 62px; height: 35px; transform: scale(0.8); }
/* Hide default HTML checkbox */
.switch input { opacity: 1; width: 0; height: 0;
}
/* The slider */
.slider { position: absolute; cursor: pointer; top: 0; left: 0; right: 0; bottom: 0px;
background: var(--color-bg); transition: .4s; border-radius: 30px; border: 1px solid var(--color-borders);
}
.slider:before { position: absolute; content: "";
height: 1.9em; width: 1.9em; border-radius: 16px; left: 1.2px; top: 0; bottom: 0
background-color: var(--color-bg); box-shadow: 0 2px 5px var(--color-accent-hover); transition: .4s;
}
input:checked + .slider { background-color: var(--color-accent); border: 1px solid transparent;
}
input:checked + .slider:before { transform: translateX(1.5em); background-color: var(--color-accent-focus); }

View File

@@ -0,0 +1,492 @@
"use strict";
class App {
lang;
tls;
config;
parserList;
uploadableFiles;
wrap;
actionsArea;
aside;
main;
generateTextElement(type, value) {
const elem = document.createElement(type);
elem.textContent = value;
return elem;
}
generateLabelInput(id, labelText, explanationText, type, value) {
const output = document.createElement("div");
output.classList.add("label-input");
const label = document.createElement("label");
label.textContent = labelText;
label.setAttribute("for", id);
output.appendChild(label);
const input = document.createElement("input");
input.id = id;
input.type = "text";
input.value = value;
output.appendChild(input);
const help = document.createElement("span");
help.classList.add("icon-help");
help.title = explanationText;
output.appendChild(help);
return {'wrap': output, 'input': input};
}
async generateSidebar() {
this.aside.appendChild(this.generateTextElement("h2", this.tls.settings));
// Set up fields
const instanceLinkArea = this.generateLabelInput("instance-link",
this.tls.instance_link, this.tls.instance_link_explica, "url", this.config.settings.instance);
this.aside.appendChild(instanceLinkArea.wrap);
const usernameArea = this.generateLabelInput("username",
this.tls.username, this.tls.username_explica, "text", this.config.settings.username);
this.aside.appendChild(usernameArea.wrap);
const mailArea = this.generateLabelInput("mail",
this.tls.mail, this.tls.mail_explica, "email", this.config.settings.mail);
this.aside.appendChild(mailArea.wrap);
const institutionIdArea = this.generateLabelInput("institutionId",
this.tls.institution_id, this.tls.institution_id_explica, "number", this.config.settings.institution_id);
this.aside.appendChild(institutionIdArea.wrap);
const tokenArea = this.generateLabelInput("token",
this.tls.token, this.tls.token_explica, "text", this.config.settings.token);
this.aside.appendChild(tokenArea.wrap);
const parserDiv = document.createElement("div");
parserDiv.classList.add("label-input");
const parserLabel = document.createElement("label");
parserLabel.setAttribute("for", "parserSelect");
parserLabel.textContent = this.tls.parser;
parserDiv.appendChild(parserLabel);
const parserSelect = document.createElement("select");
parserSelect.id = "parserSelect";
for (const entry of this.parserList) {
const opt = document.createElement("option");
opt.value = entry.title;
opt.textContent = entry.title;
if (entry.title === this.config.settings.parser) {
opt.setAttribute("selected", "selected");
}
parserSelect.appendChild(opt);
}
parserDiv.appendChild(parserSelect);
const parserHelp = document.createElement("span");
parserHelp.classList.add("icon-help");
parserHelp.title = this.tls.parser_explica;
parserDiv.appendChild(parserHelp);
this.aside.appendChild(parserDiv);
const uploadFolderArea = this.generateLabelInput("upload-folder",
this.tls.upload_directory, this.tls.upload_directory_explica, "text", this.config.settings.upload_directory);
this.aside.appendChild(uploadFolderArea.wrap);
const visibleDivOuter = document.createElement("div");
visibleDivOuter.classList.add("label-input");
const visibleLabel = document.createElement("label");
visibleLabel.setAttribute("for", "visible");
visibleLabel.textContent = this.tls.publish_immediately;
visibleDivOuter.appendChild(visibleLabel);
const visibleDiv = document.createElement("div");
const visibleSwitch = document.createElement("div");
visibleSwitch.classList.add("switch");
const visibleInput = document.createElement("input");
visibleInput.type = "checkbox";
if (this.config.settings.visible === true) {
visibleInput.checked = "checked";
console.log(visibleInput);
}
visibleSwitch.appendChild(visibleInput);
const visibleSlider = document.createElement("span");
visibleSlider.classList.add("slider");
visibleSlider.addEventListener('click', function() {
visibleInput.checked = !visibleInput.checked;
console.log(visibleInput);
});
visibleSwitch.appendChild(visibleSlider);
visibleDiv.appendChild(visibleSwitch);
visibleDivOuter.appendChild(visibleDiv);
const visibleHelp = document.createElement("span");
visibleHelp.classList.add("icon-help");
visibleHelp.title = this.tls.publish_immediately_explica;
visibleDivOuter.appendChild(visibleHelp);
this.aside.appendChild(visibleDivOuter);
// Additional settings
this.aside.appendChild(this.generateTextElement("h3", this.tls.additional_settings));
const addSettingsArea = document.createElement("div");
const app = this;
function generateAdditionalSettingsField(key, value) {
const line = document.createElement("div");
line.classList.add("label-input");
const newKey = document.createElement("input");
newKey.placeholder = app.tls.add_settings_key;
newKey.required = "required";
newKey.value = key;
line.appendChild(newKey);
const newVal = document.createElement("input");
newVal.placeholder = app.tls.add_settings_value;
newVal.required = "required";
newVal.value = value;
line.appendChild(newVal);
const removeB = document.createElement("button");
removeB.textContent = " x ";
removeB.title = app.tls.remove;
line.appendChild(removeB);
removeB.addEventListener('click', function() {
line.parentElement.removeChild(line);
});
return line;
}
if (this.config.settings.settings != undefined && this.config.settings.settings != null && this.config.settings.settings.length != 0) {
for (let settingKey in this.config.settings.settings) {
addSettingsArea.appendChild(generateAdditionalSettingsField(settingKey, this.config.settings.settings[settingKey]));
}
}
this.aside.appendChild(addSettingsArea);
const newSettingLine = document.createElement("form");
newSettingLine.classList.add("label-input");
const newKey = document.createElement("input");
newKey.placeholder = this.tls.add_settings_key;
newKey.required = "required";
newSettingLine.appendChild(newKey);
const newVal = document.createElement("input");
newVal.placeholder = this.tls.add_settings_value;
newVal.required = "required";
newSettingLine.appendChild(newVal);
const newSubmitB = document.createElement("button");
newSubmitB.type = "submit";
newSubmitB.textContent = " > ";
newSubmitB.title = app.tls.add;
newSettingLine.appendChild(newSubmitB);
newSettingLine.addEventListener('submit', function(e) {
e.preventDefault();
e.stopPropagation();
addSettingsArea.appendChild(generateAdditionalSettingsField(newKey.value, newVal.value));
newKey.value = "";
newVal.value = "";
});
this.aside.appendChild(newSettingLine)
const updateB = document.createElement("button");
updateB.textContent = this.tls.save_update;
updateB.classList.add("updateB");
this.aside.appendChild(updateB);
updateB.addEventListener('click', async function() {
let toSave = {
instance: instanceLinkArea.input.value,
username: usernameArea.input.value,
mail: mailArea.input.value,
token: tokenArea.input.value,
institution_id: parseInt(institutionIdArea.input.value),
parser: parserSelect.value,
upload_directory: uploadFolderArea.input.value,
visible: visibleInput.checked,
settings: {}
};
for (const l of addSettingsArea.children) {
toSave.settings[l.children[0].value] = l.children[1].value;
}
const response = await window.fetch('/update-settings', {
method: 'POST', cache: 'no-cache',
credentials: 'same-origin',
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
},
body: "settings=" + encodeURIComponent(JSON.stringify(toSave, null, 2))
});
if (!response.ok) {
console.log("Failed to save");
window.alert("Failed to store.");
}
const json = await response.json();
if (json.success === true) {
location.reload();
}
else {
alert("Error:\n" + json.error);
}
});
}
generateMain() {
this.main.appendChild(this.generateTextElement("h2", this.tls.uploadable_files));
// Metadata
const metadataSec = document.createElement("section");
metadataSec.appendChild(this.generateTextElement("h3", this.tls.metadata_files));
const metadataList = document.createElement("div");
metadataList.classList.add("upload-file-list");
for (const filename of this.uploadableFiles.metadata) {
const fileline = document.createElement("span");
fileline.classList.add("icon-file");
fileline.textContent = filename;
metadataList.appendChild(fileline);
}
metadataSec.appendChild(metadataList);
this.main.appendChild(metadataSec);
// Media
const mediaSec = document.createElement("section");
mediaSec.appendChild(this.generateTextElement("h3", this.tls.media_files));
const mediaList = document.createElement("div");
mediaList.classList.add("upload-file-list");
for (const filename of this.uploadableFiles.media_files) {
const fileline = document.createElement("span");
fileline.classList.add("icon-file");
fileline.textContent = filename;
mediaList.appendChild(fileline);
}
mediaSec.appendChild(mediaList);
this.main.appendChild(mediaSec);
}
manageUpload() {
const transferOverlay = document.createElement("div");
transferOverlay.id = "transfer-overlay";
transferOverlay.classList.add("transfer-overlay");
const hl = document.createElement("div");
hl.classList.add("transfer-overlay-hl");
hl.appendChild(this.generateTextElement("h3", this.tls.upload));
const closeB = document.createElement("span");
closeB.classList.add("overlay-close");
hl.appendChild(closeB);
closeB.addEventListener('click', function() { location.reload(); });
// closeB.addEventListener('click', function() { transferOverlay.parentElement.removeChild(transferOverlay); });
transferOverlay.appendChild(hl);
const transferMsgs = document.createElement("div");
transferMsgs.classList.add("transfer-msgs");
transferOverlay.appendChild(transferMsgs);
this.wrap.appendChild(transferOverlay);
const iframe = document.createElement("iframe");
iframe.src = "/trigger-upload";
transferOverlay.appendChild(iframe);
}
generateActionsMenu() {
if (this.uploadableFiles.metadata.length === 0 && this.uploadableFiles.media_files.length === 0) {
return false;
}
const uploadTrigger = document.createElement("span");
uploadTrigger.textContent = this.tls.upload;
this.actionsArea.appendChild(uploadTrigger);
const app = this;
uploadTrigger.addEventListener('click', function() {
app.manageUpload();
}, {once: true});
// Handle next auto upload
const nextAutoUpload = document.createElement("span");
const startTimeStr = this.tls.next_scheduled_check + ": ";
this.actionsArea.appendChild(nextAutoUpload);
const options1 = {
year: 'numeric',
month: '2-digit',
day: '2-digit',
hour: '2-digit',
minute: '2-digit',
};
const dateTimeFormat1 = new Intl.DateTimeFormat(this.lang, options1);
const nextUploadTime = Date.parse(this.config.next_auto_upload);
const now = new Date();
setInterval(function() {
const diffMs = nextUploadTime - now;
const diffMins = Math.round((diffMs % 86400000) / 60000); // minutes
nextAutoUpload.textContent = startTimeStr + dateTimeFormat1.format(nextUploadTime) + " (in " + diffMins + " mins)";
}, 60);
// Version
const vSpan = document.createElement("a");
vSpan.textContent = this.tls.version + " " + this.config.version;
if (this.config.update_available === true) {
vSpan.style.color = "red";
vSpan.href = "https://gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/releases";
}
this.actionsArea.appendChild(vSpan)
}
// Provides a sidebar entry for describing the parser
generateParserDesc() {
for (const parser of this.parserList) {
if (parser.title == this.config.settings.parser) {
const aside = document.createElement("div");
aside.classList.add("sec-aside");
aside.appendChild(this.generateTextElement("h3", this.tls.parser + ": " + parser.title));
const div = document.createElement("div");
div.classList.add("parser-desc");
const desc = document.createElement("p");
desc.textContent = parser.comment;
div.appendChild(desc);
aside.appendChild(div);
this.wrap.appendChild(aside);
break;
}
}
}
render() {
if (this.config.setup_required === true) {
wrap.classList.add("setup-required");
this.generateSidebar();
}
else {
this.generateSidebar();
this.generateMain();
this.generateActionsMenu();
this.generateParserDesc();
}
}
setupFromApis() {
const app = this;
let done = 0;
function wrapFinish() {
if (done === 4) {
app.render();
}
}
window.fetch('/get-settings', {method: 'GET', cache: 'no-cache', credentials: 'same-origin'})
.then(function(response) { return response.json(); })
.then(function(elements) {
app.config = elements;
done++;
wrapFinish();
});
window.fetch('/list-parsers', {method: 'GET', cache: 'no-cache', credentials: 'same-origin'})
.then(function(response) { return response.json(); })
.then(function(elements) {
app.parserList = elements;
done++;
wrapFinish();
});
window.fetch('/list-uploadable', {method: 'GET', cache: 'no-cache', credentials: 'same-origin'})
.then(function(response) { return response.json(); })
.then(function(elements) {
app.uploadableFiles = elements;
done++;
wrapFinish();
});
window.fetch('/get-translations?lang=' + encodeURIComponent(this.lang), {method: 'GET', cache: 'no-cache', credentials: 'same-origin'})
.then(function(response) { return response.json(); })
.then(function(elements) {
app.tls = elements.webdav_uploader;
done++;
wrapFinish();
});
}
constructor() {
this.wrap = document.getElementById("wrap");
this.lang = navigator.language.substring(0, 2)
if (["de", "en"].includes(this.lang) === false) {
this.lang = "en";
}
this.aside = document.createElement("aside");
this.main = document.createElement("main");
this.actionsArea = document.getElementById("actions");
this.wrap.appendChild(this.aside);
this.wrap.appendChild(this.main);
this.setupFromApis();
}
}
new App();

55
src/webui/headers.go Normal file
View File

@@ -0,0 +1,55 @@
package webui
import (
"net/http"
)
// Sets generally useful HTTP headers, e.g. limiting frame-options.
func setDefaultHeaders(w http.ResponseWriter) {
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("X-Frame-Options", "SAMEORIGIN")
w.Header().Set("X-XSS-Protection", "1; mode=block")
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Expose-Headers", "Content-Type")
// w.Header().Set("Strict-Transport-Security", "max-age=63072000")
}
// Sets the HTTP headers expected to be returned when serving HTML pages.
func setHeadersForHtml(w http.ResponseWriter) {
setDefaultHeaders(w)
w.Header().Set("Content-Security-Policy", "default-src 'self'; font-src 'self'; object-src 'none'; frame-src 'self'; frame-ancestors 'none'; base-uri 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'")
}
// Sets the HTTP headers expected to be returned when serving JSON.
func setHeadersForJson(w http.ResponseWriter) {
setDefaultHeaders(w)
w.Header().Set("Content-Type", "application/json")
}
// Sets the HTTP headers expected to be returned when serving CSS files.
func setHeadersForCss(w http.ResponseWriter) {
setDefaultHeaders(w)
w.Header().Set("Content-Type", "text/css")
}
// Sets the HTTP headers expected to be returned when serving JS files.
func setHeadersForJs(w http.ResponseWriter) {
setDefaultHeaders(w)
w.Header().Set("Content-Type", "text/javascript")
}
// Sets the HTTP headers expected to be returned when serving SVG files.
func setHeadersForSvg(w http.ResponseWriter) {
setDefaultHeaders(w)
w.Header().Set("Content-Type", "image/svg+xml")
}
func setHeadersForEventStream(w http.ResponseWriter) {
setDefaultHeaders(w)
w.Header().Set("X-Accel-Buffering", "no")
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
}

View File

@@ -0,0 +1,39 @@
package webui
import (
"encoding/json"
"fmt"
"net/http"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/meta"
)
type getSettingsApiResponse struct {
Version string `json:"version"`
SetupRequired bool `json:"setup_required"`
Settings configloader.MDWebDavUploaderConfig `json:"settings"`
NextAutoUpload string `json:"next_auto_upload"`
UpdateAvailable bool `json:"update_available"`
}
// Generates the API output for listing current settings.
func serveApiGetSettings(w http.ResponseWriter, r *http.Request) {
var nextAutoUpOut string
nextAutoUp, err := scheduledUpload.NextRun()
if err != nil {
nextAutoUpOut = "";
} else {
nextAutoUpOut = nextAutoUp.Format("2006-01-02 15:04:05")
}
setHeadersForJson(w)
output := getSettingsApiResponse{SetupRequired: setupRequired, Settings: config, Version: meta.GetVersion(), NextAutoUpload: nextAutoUpOut, UpdateAvailable: updateAvailable}
outputJson, encodeErr := json.Marshal(output)
if encodeErr != nil {
panic("Failed to create JSON")
}
fmt.Fprint(w, string(outputJson))
}

View File

@@ -0,0 +1,32 @@
package webui
import (
_ "embed"
"fmt"
"net/http"
)
type tlRaw struct {
webdav_uploader map[string]string `json:"webdav_uploader"`
}
//go:embed translation-json/webdav-uploader/de/webdav_uploader.json
var rawTlsDe string
//go:embed translation-json/webdav-uploader/en/webdav_uploader.json
var rawTlsEn string
// Retrieves translation variables in the given language.
func serveApiGetTranslations(w http.ResponseWriter, r *http.Request) {
var rawTls string
if r.URL.Query().Get("lang") == "de" {
rawTls = rawTlsDe
} else {
rawTls = rawTlsEn
}
setHeadersForJson(w)
fmt.Fprint(w, rawTls)
}

View File

@@ -0,0 +1,22 @@
package webui
import (
"encoding/json"
"fmt"
"net/http"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
)
// Provides the JSON API for listing all available parsers.
func serveApiListParsers(w http.ResponseWriter, r *http.Request) {
setHeadersForJson(w)
outputJson, encodeErr := json.Marshal(configloader.ListParsers())
if encodeErr != nil {
panic("Failed to create JSON")
}
fmt.Fprint(w, string(outputJson))
}

View File

@@ -0,0 +1,36 @@
package webui
import (
"encoding/json"
"fmt"
"net/http"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/uploadsrcdir"
)
type listUploadableApiResponse struct {
Metadata []string `json:"metadata"`
MediaFiles []string `json:"media_files"`
}
// Provides the JSON API for listing all metadata and media files that
// are currently listed to be uploaded.
func serveApiListUploadable(w http.ResponseWriter, r *http.Request) {
setHeadersForJson(w)
var output listUploadableApiResponse
if setupRequired == true {
output = listUploadableApiResponse{Metadata: []string{}, MediaFiles: []string{}}
} else {
uploadableMetadata, uploadableMedia := uploadsrcdir.GetUploadableFiles(config)
output = listUploadableApiResponse{Metadata: uploadableMetadata, MediaFiles: uploadableMedia}
}
outputJson, encodeErr := json.Marshal(output)
if encodeErr != nil {
panic("Failed to create JSON")
}
fmt.Fprint(w, string(outputJson))
}

View File

@@ -0,0 +1,55 @@
package webui
import (
"fmt"
"net/http"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/uploadsrcdir"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/webdavupload"
)
// Provides the API logic for uploading files.
func serveApiTriggerUpload(w http.ResponseWriter, r *http.Request) {
fmt.Println("Upload requested")
setHeadersForEventStream(w)
w.(http.Flusher).Flush()
fmt.Fprint(w, "Getting list of uploadable files\n")
w.(http.Flusher).Flush()
uploadableMetadata, uploadableMedia := uploadsrcdir.GetUploadableFiles(config)
// If there are no files to upload, do nothing
if len(uploadableMetadata) == 0 && len(uploadableMedia) == 0 {
return
}
fmt.Fprint(w, "Connecting to WebDAV share\n")
w.(http.Flusher).Flush()
c := webdavupload.GetWebdavClient(config)
fmt.Fprint(w, "Connected\n")
w.(http.Flusher).Flush()
if webdavupload.CheckRemoteIsFree(c) == false {
fmt.Println("The remote is currently occupied (very recent files, or an import config is currently waiting to be processed, are present).")
return
}
// Upload the files
if len(uploadableMetadata) > 0 {
fmt.Fprint(w, "Uploading metadata files\n")
w.(http.Flusher).Flush()
webdavupload.UploadMetadataFiles(c, w, uploadableMetadata)
webdavupload.BatchUnlink(w, uploadableMetadata)
}
if len(uploadableMedia) > 0 {
fmt.Fprint(w, "Uploading media files\n")
webdavupload.UploadMediaFiles(c, w, uploadableMedia)
webdavupload.BatchUnlink(w, uploadableMedia)
}
fmt.Fprint(w, "Generating and uploading import configuration\n")
webdavupload.SetImportConfigToTheRemote(c, config)
fmt.Fprint(w, "DONE\n")
w.(http.Flusher).Flush()
}

View File

@@ -0,0 +1,59 @@
package webui
import (
"encoding/json"
"fmt"
"net/http"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
)
type updateSettingsResponse struct {
Success bool `json:"success"`
ValidationError string `json:"error"`
}
// Provides the API for updating the general configuration.
func serveApiUpdateSettings(w http.ResponseWriter, r *http.Request) {
setHeadersForJson(w)
// Parse POST data
parseErr := r.ParseForm()
if parseErr != nil {
fmt.Println("Failed to load form data")
return
}
// Load new config from a POST variable 'settings' that provides the full
// settings as a JSON-encoded string.
rawInputConfig := r.PostForm.Get("settings")
var newConfig configloader.MDWebDavUploaderConfig
unmarshalErr := json.Unmarshal([]byte(rawInputConfig), &newConfig)
if unmarshalErr != nil {
fmt.Fprint(w, "{\"success\": false, \"error\": \"Invalid input JSON\"}")
return
}
// Attempt to store the new settings. Validation takes place while storing
var output updateSettingsResponse
validationErr := configloader.StoreConfigToFile(newConfig, "")
if validationErr == nil {
output = updateSettingsResponse{Success: true, ValidationError: ""}
// Reload config
config, setupRequired, _ = configloader.LoadFromFile("")
} else {
output = updateSettingsResponse{Success: false, ValidationError: validationErr.Error()}
}
// Format response in JSON, then print it
outputJson, encodeErr := json.Marshal(output)
if encodeErr != nil {
panic("Failed to create JSON")
}
fmt.Fprint(w, string(outputJson))
}

View File

@@ -0,0 +1,16 @@
package webui
import (
_ "embed"
"fmt"
"net/http"
)
//go:embed assets/index.htm
var indexHtml string
// Serves the app shell. Contents are managed via APIs and JS.
func serveAppShell(w http.ResponseWriter, r *http.Request) {
setHeadersForHtml(w)
fmt.Fprint(w, indexHtml)
}

16
src/webui/serveCss.go Normal file
View File

@@ -0,0 +1,16 @@
package webui
import (
_ "embed"
"fmt"
"net/http"
)
//go:embed assets/md-uploader.css
var webuiCss string
// Serves the style sheet of the web app.
func serveCss(w http.ResponseWriter, r *http.Request) {
setHeadersForCss(w)
fmt.Fprint(w, webuiCss)
}

16
src/webui/serveJs.go Normal file
View File

@@ -0,0 +1,16 @@
package webui
import (
_ "embed"
"fmt"
"net/http"
)
//go:embed assets/md-uploader.js
var webuiJs string
// Serves the JS for the web app.
func serveJs(w http.ResponseWriter, r *http.Request) {
setHeadersForJs(w)
fmt.Fprint(w, webuiJs)
}

16
src/webui/serveLogo.go Normal file
View File

@@ -0,0 +1,16 @@
package webui
import (
_ "embed"
"fmt"
"net/http"
)
//go:embed assets/logo-md-code-black.svg
var logoSvg string
// Serves the logo.
func serveLogo(w http.ResponseWriter, r *http.Request) {
setHeadersForSvg(w)
fmt.Fprint(w, logoSvg)
}

126
src/webui/server.go Normal file
View File

@@ -0,0 +1,126 @@
package webui
import (
"fmt"
"log"
"net/http"
"os/exec"
"runtime"
"time"
"github.com/go-co-op/gocron/v2"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/cli"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/configloader"
"gitea.armuli.eu/museum-digital/museum-digital-webdav-uploader/src/meta"
)
var config configloader.MDWebDavUploaderConfig
var setupRequired bool
var scheduler gocron.Scheduler
var scheduledUpload gocron.Job
var updateAvailable bool
// Opens the application in the default browser.
func openInBrowser(port string) {
var err error
target := "localhost:" + port
switch runtime.GOOS {
case "linux":
err = exec.Command("xdg-open", target).Start()
case "windows":
err = exec.Command("rundll32", "url.dll,FileProtocolHandler", target).Start()
case "darwin":
err = exec.Command("open", target).Start()
default:
err = fmt.Errorf("unsupported platform")
}
if err != nil {
fmt.Println(err)
}
}
func scheduleUploads() {
s, err := gocron.NewScheduler()
defer func() { _ = s.Shutdown() }()
if err != nil {
panic("Failed to set up scheduler")
}
j, jobErr := s.NewJob(
gocron.DurationJob(
3 * time.Hour,
),
gocron.NewTask(
func() {
fmt.Println("Checking if an upload should be done")
fmt.Println(time.Now().Format("20060102150405"))
cli.Upload()
},
),
)
if jobErr != nil {
panic("Failed to set up scheduler job")
}
scheduledUpload = j
s.Start()
select {}
}
// Checks if a new update is available. If that can be confirmed, set updateAvailable
// to true.
func checkUpdateAvailable() {
output, err := meta.CheckVersionIsUpToDate()
// If the RSS stream could not be fetched or parsed, don't say anything
if err != nil || output.Title == "" {
updateAvailable = false
} else {
updateAvailable = true
}
}
// Sets up the server and manages routing.
func Run() {
checkUpdateAvailable()
config, setupRequired, _ = configloader.LoadFromFile("")
// Bind callable URLs
http.HandleFunc("/", serveAppShell) // Serve app shell
http.HandleFunc("/get-settings", serveApiGetSettings) // API for listing current settings
http.HandleFunc("/get-translations", serveApiGetTranslations) // API for listing current settings
http.HandleFunc("/list-uploadable", serveApiListUploadable) // API to list uploadable files
http.HandleFunc("/list-parsers", serveApiListParsers) // API to list available parsers.
http.HandleFunc("/update-settings", serveApiUpdateSettings) // API for updating settings
http.HandleFunc("/trigger-upload", serveApiTriggerUpload) // Serve page for specific files
http.HandleFunc("/md-uploader.css", serveCss) // Main CSS file.
http.HandleFunc("/md-uploader.js", serveJs) // Main JS file.
http.HandleFunc("/logo.svg", serveLogo) // Logo.
port := "8080"
go func() {
// Set port to listen on
fmt.Println("Server started at port " + string(port))
err := http.ListenAndServe(":"+port, nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
}();
go func() {
time.Sleep(5)
openInBrowser(port)
}();
go scheduleUploads()
select{}
}