1. Introduction

In this tutorial, we go over TASTy (Typed Abstract Syntax Trees), the importance in the compiler toolchain, the compiler directives we use to manipulate TASTy files, and compile-time TASTy reflection. TASTy files (.tasty) are generated and used by the compiler to store program information not available in low-level .class files designed for the JVM.

During the transition from Scala 2 to Scala 3, a new compiler was developed from scratch, called Dotty, utilizing a new TASTy format. Understanding Scala 3 TASTy files will give us a better understanding of compiler internals and the abstract representation of our programs in Scala 3.

2. TASTy Files

When we compile a Scala program, TASTy files are generated as an abstract representation of the source code, including the syntax, typing, and documentation.

Let’s consider a simple program:

final val TASTy: String = "TASTy"

@main def main: Unit = s"Mmm, $TASTy"

If we place the program in the file TASTy.scala and compile it:

$ scalac TASTy.scala
$ ls
 main.class              'TASTy$package.tasty'
 main.tasty              TASTy.scala
 'TASTy$package$.class'  'TASTy$package.class'

We see two .tasty files generated.

2.1. Compiling TASTy Files

We can compile .tasty files into .class files without source code using the -from-tasty directive of the compiler:

$ scalac -from-tasty main.tasty

2.2. Decompiling TASTy Files

We can decompile .tasty files back into Scala source code using the compiler:

$ scalac -decompile 'TASTy$package.tasty'
/** Decompiled from ./TASTy$package.tasty */
@scala.annotation.internal.SourceFile("TASTy.scala") object TASTy$package {
  final val TASTy: scala.Predef.String = "TASTy"
  @scala.main def main: scala.Unit = {
    _root_.scala.StringContext.apply("Mmm, ", "").s(TASTy$package.TASTy)
    ()
  }
}
$ scalac -decompile 'main.tasty'
/** Decompiled from ./main.tasty */
@scala.annotation.internal.SourceFile("TASTy.scala") final class main() {
  def main(args: scala.Array[java.lang.String]): scala.Unit = try TASTy$package.main catch {
    case error: scala.util.CommandLineParser.ParseError =>
      scala.util.CommandLineParser.showError(error)
  }
}

This is what the code looks like without syntactic sugar.

2.3. Print Tasty Directive

We can also print a TASTy file using a different compiler directive:

$ scalac -print-tasty 'TASTy$package.tasty'
--------------------------------------------------------------------------------
TASTy$package.tasty
--------------------------------------------------------------------------------
Names:
0: ASTs
1: <empty>
2: TASTy$package
3: TASTy$package[ModuleClass]
4: <init>
5: <init>[Signed Signature(List(),TASTy$package$) @<init>]
6: java
7: lang
8: java[Qualified . lang]
9: Object
10: java[Qualified . lang][Qualified . Object]
11: <init>[Signed Signature(List(),java.lang.Object) @<init>]
12: _
13: Unit
14: scala
15: writeReplace
16: AnyRef
17: runtime
18: scala[Qualified . runtime]
19: ModuleSerializationProxy
20: scala[Qualified . runtime][Qualified . ModuleSerializationProxy]
21: Class
22: java[Qualified . lang][Qualified . Class]
23: <init>[Signed Signature(List(java.lang.Class),scala.runtime.ModuleSerializationProxy) @<init>]
24: TASTy
25: String
26: Predef
27: main
28: s
29: java[Qualified . lang][Qualified . String]
30: collection
31: scala[Qualified . collection]
32: immutable
33: scala[Qualified . collection][Qualified . immutable]
34: Seq
35: scala[Qualified . collection][Qualified . immutable][Qualified . Seq]
36: s[Signed Signature(List(scala.collection.immutable.Seq),java.lang.String) @s]
37: apply
38: StringContext
39: scala[Qualified . StringContext]
40: apply[Signed Signature(List(scala.collection.immutable.Seq),scala.StringContext) @apply]
41: _root_
42: StringContext[ModuleClass]
43: Mmm, 
44: 
45: <repeated>
46: Any
47: scala[Qualified . main]
48: <init>[Signed Signature(List(),scala.main) @<init>]
49: SourceFile
50: annotation
51: scala[Qualified . annotation]
52: internal
53: scala[Qualified . annotation][Qualified . internal]
54: scala[Qualified . annotation][Qualified . internal][Qualified . SourceFile]
55: <init>[Signed Signature(List(java.lang.String),scala.annotation.internal.SourceFile) @<init>]
56: TASTy.scala
57: Positions
58: Comments

The first we see are names, which are indexed and unique identifiers found in our program. Sometimes the name of our value is changed, like when private members need to be accessed from inheriting classes or in nested classes.

Then we see the tree structure of the file:

Trees:
start = Addr(0), base = 456, current = Addr(0), end = Addr(217)
217 bytes of AST, base = Addr(0)

     0: PACKAGE(214)
     3:   TERMREFpkg 1 [<empty>]
     5:   VALDEF(20) 2 [TASTy$package]
     8:     IDENTtpt 3 [TASTy$package[ModuleClass]]
    10:       TYPEREFsymbol 27
    12:         THIS
    13:           TYPEREFpkg 1 [<empty>]
    15:     APPLY(8)
    17:       SELECTin(6) 5 [<init>[Signed Signature(List(),TASTy$package$) @<init>]]
    20:         NEW
    21:           SHAREDterm 8
    23:         SHAREDtype 10
    25:     OBJECT
    26:     SYNTHETIC
    27:   TYPEDEF(187) 3 [TASTy$package[ModuleClass]]
    31:     TEMPLATE(161)
    34:       APPLY(10)
    36:         SELECTin(8) 11 [<init>[Signed Signature(List(),java.lang.Object) @<init>]]
    39:           NEW
    40:             TYPEREF 9 [Object]
    42:               TERMREFpkg 8 [java[Qualified . lang]]
    44:           SHAREDtype 40
    46:       SELFDEF 12 [_]
    48:         SINGLETONtpt
    49:           TERMREFsymbol 5
    51:             SHAREDtype 12
    53:       DEFDEF(6) 4 [<init>]
    56:         EMPTYCLAUSE
    57:         TYPEREF 13 [Unit]
    59:           TERMREFpkg 14 [scala]
    61:       DEFDEF(23) 15 [writeReplace]
    64:         EMPTYCLAUSE
    65:         TYPEREF 16 [AnyRef]
    67:           SHAREDtype 59
    69:         APPLY(13)
    71:           SELECTin(8) 23 [<init>[Signed Signature(List(java.lang.Class),scala.runtime.ModuleSerializationProxy) @<init>]]
    74:             NEW
    75:               TYPEREF 19 [ModuleSerializationProxy]
    77:                 TERMREFpkg 18 [scala[Qualified . runtime]]
    79:             SHAREDtype 75
    81:           CLASSconst
    82:             SHAREDtype 49
    84:         PRIVATE
    85:         SYNTHETIC
    86:       VALDEF(12) 24 [TASTy]
    89:         IDENTtpt 25 [String]
    91:           TYPEREF 25 [String]
    93:             TERMREF 26 [Predef]
    95:               SHAREDtype 59
    97:         STRINGconst 24 [TASTy]
    99:         FINAL
   100:       DEFDEF(93) 27 [main]
   103:         IDENTtpt 13 [Unit]
   105:           TYPEREF 13 [Unit]
   107:             TERMREFpkg 14 [scala]
   109:         BLOCK(63)
   111:           UNITconst
   112:           APPLY(60)
   114:             SELECTin(38) 36 [s[Signed Signature(List(scala.collection.immutable.Seq),java.lang.String) @s]]
   117:               APPLY(31)
   119:                 SELECTin(11) 40 [apply[Signed Signature(List(scala.collection.immutable.Seq),scala.StringContext) @apply]]
   122:                   SELECT 38 [StringContext]
   124:                     SELECT 14 [scala]
   126:                       TERMREFpkg 41 [_root_]
   128:                   TYPEREF 42 [StringContext[ModuleClass]]
   130:                     SHAREDtype 59
   132:                 TYPED(16)
   134:                   REPEATED(6)
   136:                     SHAREDtype 91
   138:                     STRINGconst 43 [Mmm, ]
   140:                     STRINGconst 44 []
   142:                   APPLIEDtype(6)
   144:                     TYPEREF 45 [<repeated>]
   146:                       SHAREDtype 59
   148:                     SHAREDtype 91
   150:               TYPEREF 38 [StringContext]
   152:                 SHAREDtype 59
   154:             TYPED(18)
   156:               REPEATED(8)
   158:                 TYPEREF 46 [Any]
   160:                   SHAREDtype 59
   162:                 TERMREFsymbol 86
   164:                   SHAREDtype 49
   166:               APPLIEDtype(6)
   168:                 SHAREDtype 144
   171:                 SHAREDtype 158
   174:         ANNOTATION(19)
   176:           TYPEREF 27 [main]
   178:             SHAREDtype 59
   180:           APPLY(13)
   182:             SELECTin(11) 48 [<init>[Signed Signature(List(),scala.main) @<init>]]
   185:               NEW
   186:                 IDENTtpt 27 [main]
   188:                   TYPEREF 27 [main]
   190:                     SHAREDtype 107
   192:               SHAREDtype 176
   195:     OBJECT
   196:     SYNTHETIC
   197:     ANNOTATION(18)
   199:       TYPEREF 49 [SourceFile]
   201:         TERMREFpkg 53 [scala[Qualified . annotation][Qualified . internal]]
   203:       APPLY(12)
   205:         SELECTin(8) 55 [<init>[Signed Signature(List(java.lang.String),scala.annotation.internal.SourceFile) @<init>]]
   208:           NEW
   209:             SHAREDtype 199
   212:           SHAREDtype 199
   215:         STRINGconst 56 [TASTy.scala]

And finally some other metadata:

101 position bytes:
   lines: 4
   line sizes: 33, 0, 37, 0
   positions:
         0: 0 .. 72
         5: 0 .. 0
         8: 0 .. 0
        27: 0 .. 72
        31: 0 .. 72
        40: 0 .. 0
        49: 0 .. 0
        53: 0 .. 0
        57: 0 .. 0
        61: 0 .. 0
        65: 0 .. 0
        75: 0 .. 0
        81: 0 .. 0
        86: 0 .. 33
        89: 17 .. 23
        97: 26 .. 33
       100: 35 .. 72
       103: 51 .. 55
       111: 72 .. 72
       114: 58 .. 72
       126: 60 .. 60
       136: 71 .. 71
       138: 60 .. 65
       140: 71 .. 71
       142: 71 .. 71
       158: 71 .. 71
       162: 66 .. 71
       166: 71 .. 71
       185: 35 .. 40
       186: 36 .. 40
       203: 0 .. 72
       209: 0 .. 0
       215: 0 .. 0

 source paths:
         0: TASTy.scala


 0 comment bytes:

4. TASTy Inspection

If accessing the TASTy structure through program reflection is more convenient than in a terminal, the Scala compiler team has a library that allows direct inspection of the TASTy structure called scala3-tasty-inspector. We can add the library to an SBT project with this artifact:

libraryDependencies += "org.scala-lang" %% "scala3-tasty-inspector" % scala3Version

Or we can use the official pre-made template for TASTy inspection:

$ sbt new scala/scala3-tasty-inspector.g8

To test it, we can define a package:

package lib

object Greetings {
  def hello(name: String): String = "Hello " + name
}

Then we define another package we use to inspect its structure:

object Inspector:

  def showCodeOf(tastyFile: String): String =
    var tastyStr: String = null
    val inspector = new Inspector:
      def inspect(using Quotes)(tastys: List[Tasty[quotes.type]]): Unit = 
        import quotes.reflect.*
        for tasty <- tastys do
          val projectDir = new java.io.File( "." ).getCanonicalPath() + java.io.File.separator // To cleanup the paths in @SourceFile
          tastyStr = tasty.ast.show.replace(projectDir, "")
    TastyInspector.inspectTastyFiles(List(tastyFile))(inspector)
    tastyStr

@main def run =
  val tastyFile = "lib/target/scala-3.4.2/classes/lib/Greetings.tasty"
  val tastyContents = Inspector.showCodeOf(tastyFile)
  println(tastyContents)

When we run the inspector package, we print the decompiled contents of the lib package:

[info] running inspector.run 
package lib {
  @scala.annotation.internal.SourceFile("lib/src/main/scala/lib/Greetings.scala") object Greetings {
    def hello(name: scala.Predef.String): scala.Predef.String = "Hello ".+(name)
  }
}

The Inspector gives us access to the TASTy file as an AST. Tasty is a trait with two abstract methods:

trait Tasty[Q <: Quotes & Singleton]:
  def ast: Tree
  def path: String

Tree models the AST of any Scala 3 program:

TASTy Inspector AST

We can inspect and manipulate the AST accordingly.

5. Macros

Scala metaprogramming has been revamped with new macro capabilities built on top of TASTy inspection. Instead of inspecting .tasty files, we use quoted expressions and types from the quoted package:

import scala.quoted.*

object PowerMacro:
  // The macro that unrolls the computation of powers and then generates the expression
  inline def showAsPowerTerm(inline x: Double, n: Int): String = ${ showAsTermImpl('x, 'n) }

  // The actual implementation of the macro
  private def showAsTermImpl(x: Expr[Double], n: Expr[Int])(using Quotes): Expr[String] =
    import quotes.reflect.*

    n.value match
      case Some(num) =>
        val powerExpr = unrolledPowerCode(x, num)
        Expr(powerExpr.asTerm.toString) // Ensures that the asTerm method call is evaluated at compile-time
      case None => 
        '{ "Error: 'n' must be a known constant at compile time." }

  // Helper method to unroll the power computation
  def unrolledPowerCode(x: Expr[Double], n: Int)(using Quotes): Expr[Double] =
    if n == 0 then '{ 1.0 }
    else if n == 1 then x
    else '{ $x * ${ unrolledPowerCode(x, n-1) } }

We can then use showAsPowerTerm to generate the unrolledPowerCode AST at compile-time:

@main def runMacro(): Unit =
  val expr: String = PowerMacro.showAsPowerTerm(2.0, 3)
  println(expr)

The output is the term structure generated by our macro at compile-time:

Inlined(
  Ident(PowerMacro$), 
  List(), 
  Apply(
    Select(
      Inlined(
        EmptyTree, 
        List(), 
        Inlined(
          EmptyTree, 
          List(), 
          Literal(Constant(2.0))
        )
      ),
      '*'
    ),
    List(
      Inlined(
        EmptyTree, 
        List(), 
        Inlined(
          Ident(PowerMacro$), 
          List(), 
          Apply(
            Select(
              Inlined(
                EmptyTree, 
                List(), 
                Inlined(
                  EmptyTree, 
                  List(), 
                  Literal(Constant(2.0))
                )
              ),
              '*'
            ),
            List(
              Inlined(
                EmptyTree, 
                List(), 
                Inlined(
                  EmptyTree, 
                  List(), 
                  Literal(Constant(2.0))
                )
              )
            )
          )
        )
      )
    )
  )
)

TASTy reflection on quotations allows us to inspect and manipulate the TASTy structure of our program at compile-time.

6. Conclusion

In this article, we reviewed the importance of TASTy files in the Dotty (Scala 3) compiler toolchain, the basic compiler commands we use to create and understand .tasty files (including -print-tasty, -from-tasty, and -decompile), and the naming and value referencing model.

As usual, the code is available over on GitHub.