Options
All
  • Public
  • Public/Protected
  • All
Menu

Class ArtistScraper

Manages the scraping and storage of artists from Rate Your Music.

For more information on class properties, see corresponding props in ArtistEntity

Hierarchy

Index

Constructors

Properties

Methods

Constructors

constructor

  • new ArtistScraper(url: string, verbose?: boolean): ArtistScraper

Properties

active

active: boolean

dataReadFromLocal

dataReadFromLocal: boolean

Scrapers always check for a local copy of the target resource (using Scraper.checkForLocalRecord) before executing a scrape from an external resource. If the resource was found (and therefore no external calls made), this is set to true.

databaseId

databaseId: number

Id of the local database record associated with this page scrape

description

description: string

A simple, human-readble description of what is being scraped. Used for logging.

disbanded

disbanded: boolean

Indicates a breakup (irrelevant for solo acts)

discographyCountRYM

discographyCountRYM: number

genreEntities

genreEntities: RymGenreEntity[]

genreScrapers

genreScrapers: GenreScraper[]

listCountRYM

listCountRYM: number

memberCount

memberCount: number

name

name: string

redis

redis: RedisHelper

used for caching failed results, to blacklist further calls

Protected repository

repository: Repository<ArtistEntity>

TypeORM repository handling all data flow in/out of database table

results

results: ResultBatch

Contains all results generated by Scraper.scrape, including recursive calls.

Protected scrapeRoot

scrapeRoot: ParseElement

Stores the DOM retrieved by scraperapi

scrapeSucceeded

scrapeSucceeded: boolean

Flag indicating a sucessful scrape, set to true after non-error-throwing call to Scraper.scrape.

showCountRYM

showCountRYM: number

soloPerformer

soloPerformer: boolean

url

url: string

External url indicating the scraper's target resource.

verbose

verbose: boolean

Used to override .env settings and force-log the output of a given scraper.

Methods

checkForLocalRecord

  • checkForLocalRecord(): Promise<boolean>

Private extractArtistName

  • extractArtistName(): void

Private extractDiscographyCount

  • extractDiscographyCount(): void

Protected extractInfo

  • extractInfo(): void

Private extractListCount

  • extractListCount(): void

Private extractMainInfoBlocks

  • extractMainInfoBlocks(): void

Private extractPastShowCount

  • extractPastShowCount(): void

getEntity

printInfo

  • printInfo(): void

printResult

  • printResult(): void

requestScrape

  • requestScrape(attempts?: number): Promise<void>

saveToLocal

  • saveToLocal(): Promise<void>

scrape

  • scrape(forceScrape?: boolean): Promise<void>

  • Entry point for initiating an asset scrape. General scrape outline/method order:

    1. Scraper.checkForLocalRecord
    2. If local entity was found, update class props and return.
    3. Scraper.requestScrape
    4. Scraper.extractInfo
    5. Scraper.scrapeDependencies
    6. Scraper.saveToLocal
    7. Update class props and return
    remarks

    This method should be considered unsafe - there are several points where this can throw errors. This is intentional, and allows easier support for relational data scraping/storage. Scraped assets may have a mixture of required and non-required dependencies, the status of which should be kept in mind when implementing Scraper.scrapeDependencies. A subclass should catch and log errors from non-required scrapes. However, errors from a required scrape should remain uncaught, so the original call to a Scraper.scrape will error out before [[Scraper.save]] is called for incomplete data.

    Parameters

    • Default value forceScrape: boolean = false

      If set to true, scrapes the external resource regardless of any existing local records

    Returns Promise<void>

Protected scrapeDependencies

  • scrapeDependencies(): Promise<void>

Protected scrapeErrorHandler

  • scrapeErrorHandler(error: Error): Promise<void>

Static scrapeDependencyArr

  • scrapeDependencyArr<T>(scrapers: T[], forceScrape?: boolean): Promise<ScrapersWithResults<T>>

Generated using TypeDoc